xenia/home
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
6fce8050ee
home/src/regex_impl.hh

673 lines
24 KiB
C++
Raw Normal View History

Regex: Add initial, exploratory work on a custom regex engine
2017-09-17 10:50:53 +02:00
#ifndef regex_impl_hh_INCLUDED
#define regex_impl_hh_INCLUDED
Regex: Add support for backward matching Regex can be compiled for backward matching instead of forward matching and the ThreadedRegexVM is able to iterate in reverse on the subject string to find the last match instead of the first.
2017-10-07 06:46:27 +02:00
#include "exception.hh"
Regex: remove use of buffer_utils.hh from regex_impl.cc
2017-10-07 04:43:21 +02:00
#include "flags.hh"
#include "ref_ptr.hh"
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
#include "unicode.hh"
#include "utf8.hh"
#include "utf8_iterator.hh"
#include "vector.hh"
Cleanup regex lookarounds implementation and reject incompatible regex Fixes #2487
2018-10-10 13:35:38 +02:00
#include "utils.hh"
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
Regex: whenever Kakoune compiles a regex, pass it to the custom impl as well That way we can see which features are missing.
2017-09-26 09:44:30 +02:00
namespace Kakoune
{
Regex: switch to custom impl, use boost for checking
2017-10-09 08:04:14 +02:00
struct regex_error : runtime_error
{
using runtime_error::runtime_error;
};
Regex: Add support for backward matching Regex can be compiled for backward matching instead of forward matching and the ThreadedRegexVM is able to iterate in reverse on the subject string to find the last match instead of the first.
2017-10-07 06:46:27 +02:00
enum class MatchDirection
{
Forward,
Backward
};
Regex: Replace generic 'Matchers' with specialized functionality Introduce CharacterClass and CharacterType Regex Op, and optimize their evaluation.
2017-11-25 11:14:15 +01:00
enum class CharacterType : unsigned char
{
None = 0,
Regex: Tweak is_ctype implementation style
2017-11-27 17:13:42 +01:00
Whitespace = 1 << 0,
HorizontalWhitespace = 1 << 1,
Word = 1 << 2,
Regex: Replace generic 'Matchers' with specialized functionality Introduce CharacterClass and CharacterType Regex Op, and optimize their evaluation.
2017-11-25 11:14:15 +01:00
Digit = 1 << 3,
Regex: Tweak is_ctype implementation style
2017-11-27 17:13:42 +01:00
NotWhitespace = 1 << 4,
NotHorizontalWhitespace = 1 << 5,
NotWord = 1 << 6,
Regex: Replace generic 'Matchers' with specialized functionality Introduce CharacterClass and CharacterType Regex Op, and optimize their evaluation.
2017-11-25 11:14:15 +01:00
NotDigit = 1 << 7
};
constexpr bool with_bit_ops(Meta::Type<CharacterType>) { return true; }
struct CharacterClass
{
struct Range { Codepoint min, max; };
Vector<Range, MemoryDomain::Regex> ranges;
CharacterType ctypes = CharacterType::None;
bool negative = false;
bool ignore_case = false;
};
bool is_character_class(const CharacterClass& character_class, Codepoint cp);
bool is_ctype(CharacterType ctype, Codepoint cp);
Regex: Introduce a Regex memory domain to track usage separately
2017-10-15 03:23:57 +02:00
struct CompiledRegex : RefCountable, UseMemoryDomain<MemoryDomain::Regex>
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
{
enum Op : char
{
Match,
Regex: Go back to instruction based search of next start The previous method, which was a bit faster in the general use case, can hit some cases where we get quadratic behaviour and very slow matching. By using an instruction, we can guarantee our complexity of O(N*M) as we will never have more than N threads (N being the instruction count) and we run the threads once per codepoint in the subject string. That slows down the general case slightly, but ensure we dont have pathological cases. This new version is much faster than the previous instruction based search because it does not use a plain `.*` searcher, but a specific, smarter instruction specialized for finding the next start if we are in the correct conditions.
2017-10-20 09:17:02 +02:00
FindNextStart,
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
Literal,
Regex: Fix support for ignore case in lookarounds
2017-10-10 05:21:21 +02:00
Literal_IgnoreCase,
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
AnyChar,
Use a dedicated vm op for dot when match-newline is false
2018-06-24 12:13:35 +02:00
AnyCharExceptNewLine,
Regex: Replace generic 'Matchers' with specialized functionality Introduce CharacterClass and CharacterType Regex Op, and optimize their evaluation.
2017-11-25 11:14:15 +01:00
Class,
CharacterType,
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
Jump,
Split_PrioritizeParent,
Split_PrioritizeChild,
Save,
LineStart,
LineEnd,
WordBoundary,
NotWordBoundary,
SubjectBegin,
SubjectEnd,
LookAhead,
NegativeLookAhead,
Regex: reorder lookaround ops, group by direction
2017-10-04 17:00:19 +02:00
LookBehind,
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
NegativeLookBehind,
Regex: Fix support for ignore case in lookarounds
2017-10-10 05:21:21 +02:00
LookAhead_IgnoreCase,
NegativeLookAhead_IgnoreCase,
LookBehind_IgnoreCase,
NegativeLookBehind_IgnoreCase,
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
};
Cleanup regex lookarounds implementation and reject incompatible regex Fixes #2487
2018-10-10 13:35:38 +02:00
enum class Lookaround : Codepoint
{
OpBegin = 0xF0000,
AnyChar = 0xF0000,
AnyCharExceptNewLine = 0xF0001,
CharacterClass = 0xF0002,
CharacterType = 0xF8000,
OpEnd = 0xFFFFF,
EndOfLookaround = static_cast<Codepoint>(-1)
};
Regex: abandon bytecode and just use a simple list of instructions Makes the code simpler.
2017-10-07 12:51:32 +02:00
struct Instruction
{
Op op;
Regex: comment the mutables in CompiledRegex::Instruction and fix their init
2017-10-15 03:34:49 +02:00
// Those mutables are used during execution
Regex: tag instructions as scheduled as well instead of searching And a few more code cleanup in the ThreadedRegexVM
2017-10-07 13:58:10 +02:00
mutable bool scheduled;
Regex: remove the need to a processed inst vector Identify each step with a counter, and check if the instruction was already processed this step. This makes the matching faster, by removing the need to maintain a vector of instructions executed this step.
2017-10-14 06:58:42 +02:00
mutable uint16_t last_step;
Regex: abandon bytecode and just use a simple list of instructions Makes the code simpler.
2017-10-07 12:51:32 +02:00
uint32_t param;
};
Regex: store the processed flag directly in CompiledRegex instructions
2017-10-07 13:08:14 +02:00
static_assert(sizeof(Instruction) == 8, "");
Regex: abandon bytecode and just use a simple list of instructions Makes the code simpler.
2017-10-07 12:51:32 +02:00
Regex: Go back to instruction based search of next start The previous method, which was a bit faster in the general use case, can hit some cases where we get quadratic behaviour and very slow matching. By using an instruction, we can guarantee our complexity of O(N*M) as we will never have more than N threads (N being the instruction count) and we run the threads once per codepoint in the subject string. That slows down the general case slightly, but ensure we dont have pathological cases. This new version is much faster than the previous instruction based search because it does not use a plain `.*` searcher, but a specific, smarter instruction specialized for finding the next start if we are in the correct conditions.
2017-10-20 09:17:02 +02:00
static constexpr uint16_t search_prefix_size = 3;
Regex: abandon bytecode and just use a simple list of instructions Makes the code simpler.
2017-10-07 12:51:32 +02:00
explicit operator bool() const { return not instructions.empty(); }
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
Regex: Introduce a Regex memory domain to track usage separately
2017-10-15 03:23:57 +02:00
Vector<Instruction, MemoryDomain::Regex> instructions;
Regex: Replace generic 'Matchers' with specialized functionality Introduce CharacterClass and CharacterType Regex Op, and optimize their evaluation.
2017-11-25 11:14:15 +01:00
Vector<CharacterClass, MemoryDomain::Regex> character_classes;
Cleanup regex lookarounds implementation and reject incompatible regex Fixes #2487
2018-10-10 13:35:38 +02:00
Vector<Lookaround, MemoryDomain::Regex> lookarounds;
Regex: Support forward and backward matching code in the same CompiledRegex No need to have two separate regexes to handle forward and backward matching, just passing RegexCompileFlags::Backward will add support for backward matching to the regex. For backward only regex, pass RegexCompileFlags::NoForward as well to disable generation of forward matching code.
2017-12-01 12:57:02 +01:00
uint32_t first_backward_inst; // -1 if no backward support, 0 if only backward, >0 if both forward and backward
uint32_t save_count;
Regex: Find potential start position using a map of valid start chars With this optimization we get close to performance parity with boost regex on the common use cases in Kakoune.
2017-10-06 07:40:27 +02:00
regex: track CompiledRegex::StartDesc in the Regex memory domain
2018-02-24 06:29:24 +01:00
struct StartDesc : UseMemoryDomain<MemoryDomain::Regex>
Regex: fix lookarounds handling when computing starting chars
2017-10-09 12:19:36 +02:00
{
Regex: Use only 128 characters in start desc and encode others as 0 Using 257 was using lots of memory for no good reason, as > 127 codepoint are not common enough to be treated specially.
2018-04-29 08:42:46 +02:00
static constexpr Codepoint count = 128;
static constexpr Codepoint other = 0;
bool map[count];
Regex: fix lookarounds handling when computing starting chars
2017-10-09 12:19:36 +02:00
};
Regex: put the other char boolean inside the general start char map
2017-10-21 04:04:08 +02:00
Regex: Support forward and backward matching code in the same CompiledRegex No need to have two separate regexes to handle forward and backward matching, just passing RegexCompileFlags::Backward will add support for backward matching to the regex. For backward only regex, pass RegexCompileFlags::NoForward as well to disable generation of forward matching code.
2017-12-01 12:57:02 +01:00
std::unique_ptr<StartDesc> forward_start_desc;
std::unique_ptr<StartDesc> backward_start_desc;
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
};
Add a debug regex command to dump regex instructions
2018-04-27 00:34:49 +02:00
String dump_regex(const CompiledRegex& program);
Regex: fix RegexCompileFlags not being an enum class
2017-10-23 11:29:03 +02:00
enum class RegexCompileFlags
Regex: switch to custom impl, use boost for checking
2017-10-09 08:04:14 +02:00
{
None = 0,
NoSubs = 1 << 0,
Regex: Support forward and backward matching code in the same CompiledRegex No need to have two separate regexes to handle forward and backward matching, just passing RegexCompileFlags::Backward will add support for backward matching to the regex. For backward only regex, pass RegexCompileFlags::NoForward as well to disable generation of forward matching code.
2017-12-01 12:57:02 +01:00
Optimize = 1 << 1,
Fix RegexCompileFlags::Backwards having the same value as Optimize That means every Optimized regex had the Backwards version compiled as well, which doubled the time it took to compile them and doubled the memory usage of regex. This should improve #2152
2018-07-19 10:34:40 +02:00
Backward = 1 << 2,
NoForward = 1 << 3,
Regex: switch to custom impl, use boost for checking
2017-10-09 08:04:14 +02:00
};
constexpr bool with_bit_ops(Meta::Type<RegexCompileFlags>) { return true; }
Regex: Support forward and backward matching code in the same CompiledRegex No need to have two separate regexes to handle forward and backward matching, just passing RegexCompileFlags::Backward will add support for backward matching to the regex. For backward only regex, pass RegexCompileFlags::NoForward as well to disable generation of forward matching code.
2017-12-01 12:57:02 +01:00
CompiledRegex compile_regex(StringView re, RegexCompileFlags flags);
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
Regex: introduce RegexExecFlags to control various behaviours
2017-10-02 16:34:57 +02:00
enum class RegexExecFlags
{
None = 0,
Search = 1 << 0,
NotBeginOfLine = 1 << 1,
NotEndOfLine = 1 << 2,
NotBeginOfWord = 1 << 3,
NotEndOfWord = 1 << 4,
Regex: take the full subject range as a parameter To allow more general look arounds out of the actual search range, pass a second range (the actual subject). This allows us to remove various flags such as PrevAvailable or NotBeginOfSubject, which are now easy to check from the subject range. Fixes #1902
2018-03-04 19:48:10 +01:00
NotInitialNull = 1 << 5,
AnyMatch = 1 << 6,
NoSaves = 1 << 7,
Regex: introduce RegexExecFlags to control various behaviours
2017-10-02 16:34:57 +02:00
};
constexpr bool with_bit_ops(Meta::Type<RegexExecFlags>) { return true; }
Use BufferCoord sentinel type for regex matching on BufferIterators BufferIterators are large-ish, and need to check the buffer pointer on comparison. Checking against a coord is just a 64 bit comparison.
2018-11-01 11:51:10 +01:00
template<typename It, typename=void>
struct SentinelType { using Type = It; };
template<typename It>
struct SentinelType<It, void_t<typename It::Sentinel>> { using Type = typename It::Sentinel; };
Regex: Add support for backward matching Regex can be compiled for backward matching instead of forward matching and the ThreadedRegexVM is able to iterate in reverse on the subject string to find the last match instead of the first.
2017-10-07 06:46:27 +02:00
template<typename Iterator, MatchDirection direction>
Regex: Make ThreadedRegexVM a proper class, define a proper interface
2017-10-06 13:30:46 +02:00
class ThreadedRegexVM
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
{
Regex: Make ThreadedRegexVM a proper class, define a proper interface
2017-10-06 13:30:46 +02:00
public:
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
ThreadedRegexVM(const CompiledRegex& program)
Regex: Add support for backward matching Regex can be compiled for backward matching instead of forward matching and the ThreadedRegexVM is able to iterate in reverse on the subject string to find the last match instead of the first.
2017-10-07 06:46:27 +02:00
: m_program{program}
Regex: small code cleanup
2017-10-09 15:56:48 +02:00
{
Regex: Support forward and backward matching code in the same CompiledRegex No need to have two separate regexes to handle forward and backward matching, just passing RegexCompileFlags::Backward will add support for backward matching to the regex. For backward only regex, pass RegexCompileFlags::NoForward as well to disable generation of forward matching code.
2017-12-01 12:57:02 +01:00
kak_assert((direction == MatchDirection::Forward and program.first_backward_inst != 0) or
(direction == MatchDirection::Backward and program.first_backward_inst != -1));
Regex: small code cleanup
2017-10-09 15:56:48 +02:00
}
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
Regex: More code tweaks and cleanups in ThreadedRegexVM
2017-10-04 14:11:15 +02:00
ThreadedRegexVM(const ThreadedRegexVM&) = delete;
Regex: Make ThreadedRegexVM a proper class, define a proper interface
2017-10-06 13:30:46 +02:00
ThreadedRegexVM& operator=(const ThreadedRegexVM&) = delete;
Regex: More code tweaks and cleanups in ThreadedRegexVM
2017-10-04 14:11:15 +02:00
Regex: Use a custom allocated buffer for Saves instead of a Vector
2017-10-04 05:14:24 +02:00
~ThreadedRegexVM()
{
for (auto* saves : m_saves)
{
for (size_t i = m_program.save_count-1; i > 0; --i)
saves->pos[i].~Iterator();
saves->~Saves();
Regex: Introduce a Regex memory domain to track usage separately
2017-10-15 03:23:57 +02:00
operator delete(saves);
Regex: Use a custom allocated buffer for Saves instead of a Vector
2017-10-04 05:14:24 +02:00
}
}
Regex: take the full subject range as a parameter To allow more general look arounds out of the actual search range, pass a second range (the actual subject). This allows us to remove various flags such as PrevAvailable or NotBeginOfSubject, which are now easy to check from the subject range. Fixes #1902
2018-03-04 19:48:10 +01:00
bool exec(Iterator begin, Iterator end,
Iterator subject_begin, Iterator subject_end,
RegexExecFlags flags)
Regex: Make ThreadedRegexVM a proper class, define a proper interface
2017-10-06 13:30:46 +02:00
{
Regex: Fix handling of ^ and $ in backward matching mode
2017-10-11 13:24:01 +02:00
if (flags & RegexExecFlags::NotInitialNull and begin == end)
return false;
constexpr bool forward = direction == MatchDirection::Forward;
Regex: take the full subject range as a parameter To allow more general look arounds out of the actual search range, pass a second range (the actual subject). This allows us to remove various flags such as PrevAvailable or NotBeginOfSubject, which are now easy to check from the subject range. Fixes #1902
2018-03-04 19:48:10 +01:00
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
if (not forward) // Flip line begin/end flags as we flipped the instructions on compilation.
flags = (RegexExecFlags)(flags & ~(RegexExecFlags::NotEndOfLine | RegexExecFlags::NotBeginOfLine)) |
Regex: Fix handling of ^ and $ in backward matching mode
2017-10-11 13:24:01 +02:00
((flags & RegexExecFlags::NotEndOfLine) ? RegexExecFlags::NotBeginOfLine : RegexExecFlags::None) |
((flags & RegexExecFlags::NotBeginOfLine) ? RegexExecFlags::NotEndOfLine : RegexExecFlags::None);
Regex: Make ThreadedRegexVM a proper class, define a proper interface
2017-10-06 13:30:46 +02:00
Regex: Go back to instruction based search of next start The previous method, which was a bit faster in the general use case, can hit some cases where we get quadratic behaviour and very slow matching. By using an instruction, we can guarantee our complexity of O(N*M) as we will never have more than N threads (N being the instruction count) and we run the threads once per codepoint in the subject string. That slows down the general case slightly, but ensure we dont have pathological cases. This new version is much faster than the previous instruction based search because it does not use a plain `.*` searcher, but a specific, smarter instruction specialized for finding the next start if we are in the correct conditions.
2017-10-20 09:17:02 +02:00
const bool search = (flags & RegexExecFlags::Search);
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
ConstArrayView<CompiledRegex::Instruction> instructions{m_program.instructions};
if (direction == MatchDirection::Forward)
instructions = instructions.subrange(0, m_program.first_backward_inst);
else
instructions = instructions.subrange(m_program.first_backward_inst);
if (not search)
instructions = instructions.subrange(CompiledRegex::search_prefix_size);
const ExecConfig config{
Use BufferCoord sentinel type for regex matching on BufferIterators BufferIterators are large-ish, and need to check the buffer pointer on comparison. Checking against a coord is just a 64 bit comparison.
2018-11-01 11:51:10 +01:00
Sentinel{forward ? begin : end},
Sentinel{forward ? end : begin},
Sentinel{forward ? subject_begin : subject_end},
Sentinel{forward ? subject_end : subject_begin},
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
flags,
instructions
};
Use BufferCoord sentinel type for regex matching on BufferIterators BufferIterators are large-ish, and need to check the buffer pointer on comparison. Checking against a coord is just a 64 bit comparison.
2018-11-01 11:51:10 +01:00
EffectiveIt start{Utf8It{
forward ? begin : end,
Sentinel{subject_begin},
Sentinel{subject_end}
}};
Regex: small code style tweak
2018-04-29 09:26:49 +02:00
if (const auto& start_desc = direction == MatchDirection::Forward ?
m_program.forward_start_desc : m_program.backward_start_desc)
Regex: use StartDesc to early out when not searching Early out as well if we do not find any potential start position.
2017-12-01 08:03:03 +01:00
{
if (search)
{
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
to_next_start(start, config.end, *start_desc);
Use BufferCoord sentinel type for regex matching on BufferIterators BufferIterators are large-ish, and need to check the buffer pointer on comparison. Checking against a coord is just a 64 bit comparison.
2018-11-01 11:51:10 +01:00
if (base(start) == config.end) // If start_desc is not null, it means we consume at least one char
Regex: use StartDesc to early out when not searching Early out as well if we do not find any potential start position.
2017-12-01 08:03:03 +01:00
return false;
}
Use BufferCoord sentinel type for regex matching on BufferIterators BufferIterators are large-ish, and need to check the buffer pointer on comparison. Checking against a coord is just a 64 bit comparison.
2018-11-01 11:51:10 +01:00
else if (base(start) != config.end and
Regex: small code style tweak
2018-04-29 09:26:49 +02:00
not start_desc->map[*start < StartDesc::count ? *start : StartDesc::other])
Regex: use StartDesc to early out when not searching Early out as well if we do not find any potential start position.
2017-12-01 08:03:03 +01:00
return false;
}
Regex: Make ThreadedRegexVM a proper class, define a proper interface
2017-10-06 13:30:46 +02:00
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
return exec_program(std::move(start), config);
Regex: Make ThreadedRegexVM a proper class, define a proper interface
2017-10-06 13:30:46 +02:00
}
ArrayView<const Iterator> captures() const
{
Use indices instead of pointers for saves/instruction in ThreadedRegexVM Performance seems unaffacted, but memory usage should be lowered as the Thread struct is 4 bytes instead of 16.
2018-04-27 00:18:04 +02:00
if (m_captures >= 0)
return { m_saves[m_captures]->pos, m_program.save_count };
Regex: Make ThreadedRegexVM a proper class, define a proper interface
2017-10-06 13:30:46 +02:00
return {};
}
private:
Regex: store saves in a copy on write structure
2017-10-03 04:54:43 +02:00
struct Saves
{
Regex: use intrusive linked list for the free saves instead of a Vector
2017-10-13 15:58:38 +02:00
union // ref count when in use, next_free when in free list
{
Regex: Use a custom 'DualThreadStack' structure to hold thread info Instead of using two vectors, we can hold both current and next threads in a single buffer, with stacks growing on each end. Benchmarking shows this to be slightly faster, and should use less memory.
2018-04-29 08:04:26 +02:00
int16_t refcount;
Use indices instead of pointers for saves/instruction in ThreadedRegexVM Performance seems unaffacted, but memory usage should be lowered as the Thread struct is 4 bytes instead of 16.
2018-04-27 00:18:04 +02:00
int16_t next_free;
Regex: use intrusive linked list for the free saves instead of a Vector
2017-10-13 15:58:38 +02:00
};
Regex: Use a custom allocated buffer for Saves instead of a Vector
2017-10-04 05:14:24 +02:00
Iterator pos[1];
Regex: store saves in a copy on write structure
2017-10-03 04:54:43 +02:00
};
Regex: Code cleanup and refactor for Saves handling
2017-10-04 13:49:16 +02:00
template<bool copy>
Use indices instead of pointers for saves/instruction in ThreadedRegexVM Performance seems unaffacted, but memory usage should be lowered as the Thread struct is 4 bytes instead of 16.
2018-04-27 00:18:04 +02:00
int16_t new_saves(Iterator* pos)
Regex: store saves in a copy on write structure
2017-10-03 04:54:43 +02:00
{
Regex: Code cleanup and refactor for Saves handling
2017-10-04 13:49:16 +02:00
kak_assert(not copy or pos != nullptr);
const auto count = m_program.save_count;
Use indices instead of pointers for saves/instruction in ThreadedRegexVM Performance seems unaffacted, but memory usage should be lowered as the Thread struct is 4 bytes instead of 16.
2018-04-27 00:18:04 +02:00
if (m_first_free >= 0)
Regex: store saves in a copy on write structure
2017-10-03 04:54:43 +02:00
{
Use indices instead of pointers for saves/instruction in ThreadedRegexVM Performance seems unaffacted, but memory usage should be lowered as the Thread struct is 4 bytes instead of 16.
2018-04-27 00:18:04 +02:00
const int16_t res = m_first_free;
Saves* save = m_saves[res];
m_first_free = save->next_free;
save->refcount = 1;
Regex: Code cleanup and refactor for Saves handling
2017-10-04 13:49:16 +02:00
if (copy)
Use indices instead of pointers for saves/instruction in ThreadedRegexVM Performance seems unaffacted, but memory usage should be lowered as the Thread struct is 4 bytes instead of 16.
2018-04-27 00:18:04 +02:00
std::copy(pos, pos + count, save->pos);
Regex: Code cleanup and refactor for Saves handling
2017-10-04 13:49:16 +02:00
else
Use indices instead of pointers for saves/instruction in ThreadedRegexVM Performance seems unaffacted, but memory usage should be lowered as the Thread struct is 4 bytes instead of 16.
2018-04-27 00:18:04 +02:00
std::fill(save->pos, save->pos + count, Iterator{});
Regex: Code cleanup and refactor for Saves handling
2017-10-04 13:49:16 +02:00
Regex: small refactor of ThreadedRegexVM::clone_saves
2017-10-03 12:23:31 +02:00
return res;
Regex: store saves in a copy on write structure
2017-10-03 04:54:43 +02:00
}
Regex: Introduce a Regex memory domain to track usage separately
2017-10-15 03:23:57 +02:00
void* ptr = operator new (sizeof(Saves) + (count-1) * sizeof(Iterator));
Regex: use intrusive linked list for the free saves instead of a Vector
2017-10-13 15:58:38 +02:00
Saves* saves = new (ptr) Saves{{1}, {copy ? pos[0] : Iterator{}}};
Regex: Code cleanup and refactor for Saves handling
2017-10-04 13:49:16 +02:00
for (size_t i = 1; i < count; ++i)
new (&saves->pos[i]) Iterator{copy ? pos[i] : Iterator{}};
m_saves.push_back(saves);
Use indices instead of pointers for saves/instruction in ThreadedRegexVM Performance seems unaffacted, but memory usage should be lowered as the Thread struct is 4 bytes instead of 16.
2018-04-27 00:18:04 +02:00
return static_cast<int16_t>(m_saves.size() - 1);
Regex: store saves in a copy on write structure
2017-10-03 04:54:43 +02:00
}
Use indices instead of pointers for saves/instruction in ThreadedRegexVM Performance seems unaffacted, but memory usage should be lowered as the Thread struct is 4 bytes instead of 16.
2018-04-27 00:18:04 +02:00
void release_saves(int16_t saves)
Regex: make m_current_threads and m_next_threads local variable of exec
2017-10-04 04:49:40 +02:00
{
Use indices instead of pointers for saves/instruction in ThreadedRegexVM Performance seems unaffacted, but memory usage should be lowered as the Thread struct is 4 bytes instead of 16.
2018-04-27 00:18:04 +02:00
if (saves >= 0 and --m_saves[saves]->refcount == 0)
Regex: use intrusive linked list for the free saves instead of a Vector
2017-10-13 15:58:38 +02:00
{
Use indices instead of pointers for saves/instruction in ThreadedRegexVM Performance seems unaffacted, but memory usage should be lowered as the Thread struct is 4 bytes instead of 16.
2018-04-27 00:18:04 +02:00
m_saves[saves]->next_free = m_first_free;
Regex: use intrusive linked list for the free saves instead of a Vector
2017-10-13 15:58:38 +02:00
m_first_free = saves;
}
Regex: make m_current_threads and m_next_threads local variable of exec
2017-10-04 04:49:40 +02:00
};
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
struct Thread
{
Use indices instead of pointers for saves/instruction in ThreadedRegexVM Performance seems unaffacted, but memory usage should be lowered as the Thread struct is 4 bytes instead of 16.
2018-04-27 00:18:04 +02:00
int16_t inst;
int16_t saves;
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
};
Regex: small code style tweak
2018-04-29 09:26:49 +02:00
using StartDesc = CompiledRegex::StartDesc;
Use BufferCoord sentinel type for regex matching on BufferIterators BufferIterators are large-ish, and need to check the buffer pointer on comparison. Checking against a coord is just a 64 bit comparison.
2018-11-01 11:51:10 +01:00
using Sentinel = typename SentinelType<Iterator>::Type;
using Utf8It = utf8::iterator<Iterator, Sentinel>;
Regex: take the full subject range as a parameter To allow more general look arounds out of the actual search range, pass a second range (the actual subject). This allows us to remove various flags such as PrevAvailable or NotBeginOfSubject, which are now easy to check from the subject range. Fixes #1902
2018-03-04 19:48:10 +01:00
using EffectiveIt = std::conditional_t<direction == MatchDirection::Forward,
Utf8It, std::reverse_iterator<Utf8It>>;
Regex: do not write the search prefix inside the program bytecode Its faster to have specialized code in the VM directly
2017-10-04 06:16:52 +02:00
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
struct ExecConfig
Regex: Only reset processed and scheduled flags on relevant instructions On big regex, reseting all those flags on all instructions for each character can become the dominant operation. Track that actual instructions index processed (the scheduled are already tracked in the next_threads vector), and only reset these.
2017-10-11 04:24:05 +02:00
{
Use BufferCoord sentinel type for regex matching on BufferIterators BufferIterators are large-ish, and need to check the buffer pointer on comparison. Checking against a coord is just a 64 bit comparison.
2018-11-01 11:51:10 +01:00
const Sentinel begin;
const Sentinel end;
const Sentinel subject_begin;
const Sentinel subject_end;
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
const RegexExecFlags flags;
ConstArrayView<CompiledRegex::Instruction> instructions;
Regex: Only reset processed and scheduled flags on relevant instructions On big regex, reseting all those flags on all instructions for each character can become the dominant operation. Track that actual instructions index processed (the scheduled are already tracked in the next_threads vector), and only reset these.
2017-10-11 04:24:05 +02:00
};
Regex: Go back to instruction based search of next start The previous method, which was a bit faster in the general use case, can hit some cases where we get quadratic behaviour and very slow matching. By using an instruction, we can guarantee our complexity of O(N*M) as we will never have more than N threads (N being the instruction count) and we run the threads once per codepoint in the subject string. That slows down the general case slightly, but ensure we dont have pathological cases. This new version is much faster than the previous instruction based search because it does not use a plain `.*` searcher, but a specific, smarter instruction specialized for finding the next start if we are in the correct conditions.
2017-10-20 09:17:02 +02:00
enum class StepResult { Consumed, Matched, Failed, FindNextStart };
Regex: Add some comments, remove supurious semicolons
2017-10-07 10:36:53 +02:00
// Steps a thread until it consumes the current character, matches or fail
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
StepResult step(EffectiveIt& pos, uint16_t current_step, Thread& thread, const ExecConfig& config)
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
{
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
const bool no_saves = (config.flags & RegexExecFlags::NoSaves);
Regex: directly store instruction pointer in Thread struct
2017-11-11 08:15:13 +01:00
auto* instructions = m_program.instructions.data();
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
while (true)
{
Use indices instead of pointers for saves/instruction in ThreadedRegexVM Performance seems unaffacted, but memory usage should be lowered as the Thread struct is 4 bytes instead of 16.
2018-04-27 00:18:04 +02:00
auto& inst = instructions[thread.inst++];
Regex: Improve comments and constify some variables Reword various comments to make some tricky parts of the regex engine easier to understand.
2018-02-24 07:21:15 +01:00
// if this instruction was already executed for this step in another thread,
// then this thread is redundant and can be dropped
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
if (inst.last_step == current_step)
Regex: store the processed flag directly in CompiledRegex instructions
2017-10-07 13:08:14 +02:00
return StepResult::Failed;
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
inst.last_step = current_step;
Regex: avoid infinite loops
2017-10-07 08:25:14 +02:00
Regex: abandon bytecode and just use a simple list of instructions Makes the code simpler.
2017-10-07 12:51:32 +02:00
switch (inst.op)
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
{
case CompiledRegex::Literal:
Use BufferCoord sentinel type for regex matching on BufferIterators BufferIterators are large-ish, and need to check the buffer pointer on comparison. Checking against a coord is just a 64 bit comparison.
2018-11-01 11:51:10 +01:00
if (base(pos) != config.end and inst.param == *pos)
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
return StepResult::Consumed;
return StepResult::Failed;
Regex: Fix support for ignore case in lookarounds
2017-10-10 05:21:21 +02:00
case CompiledRegex::Literal_IgnoreCase:
Use BufferCoord sentinel type for regex matching on BufferIterators BufferIterators are large-ish, and need to check the buffer pointer on comparison. Checking against a coord is just a 64 bit comparison.
2018-11-01 11:51:10 +01:00
if (base(pos) != config.end and inst.param == to_lower(*pos))
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
return StepResult::Consumed;
return StepResult::Failed;
case CompiledRegex::AnyChar:
return StepResult::Consumed;
Use a dedicated vm op for dot when match-newline is false
2018-06-24 12:13:35 +02:00
case CompiledRegex::AnyCharExceptNewLine:
Use BufferCoord sentinel type for regex matching on BufferIterators BufferIterators are large-ish, and need to check the buffer pointer on comparison. Checking against a coord is just a 64 bit comparison.
2018-11-01 11:51:10 +01:00
if (base(pos) != config.end and *pos != '\n')
Use a dedicated vm op for dot when match-newline is false
2018-06-24 12:13:35 +02:00
return StepResult::Consumed;
return StepResult::Failed;
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
case CompiledRegex::Jump:
Use indices instead of pointers for saves/instruction in ThreadedRegexVM Performance seems unaffacted, but memory usage should be lowered as the Thread struct is 4 bytes instead of 16.
2018-04-27 00:18:04 +02:00
thread.inst = static_cast<int16_t>(inst.param);
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
break;
case CompiledRegex::Split_PrioritizeParent:
{
Use indices instead of pointers for saves/instruction in ThreadedRegexVM Performance seems unaffacted, but memory usage should be lowered as the Thread struct is 4 bytes instead of 16.
2018-04-27 00:18:04 +02:00
if (thread.saves >= 0)
++m_saves[thread.saves]->refcount;
Regex: Use a custom 'DualThreadStack' structure to hold thread info Instead of using two vectors, we can hold both current and next threads in a single buffer, with stacks growing on each end. Benchmarking shows this to be slightly faster, and should use less memory.
2018-04-29 08:04:26 +02:00
m_threads.push_current({static_cast<int16_t>(inst.param), thread.saves});
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
break;
}
case CompiledRegex::Split_PrioritizeChild:
{
Use indices instead of pointers for saves/instruction in ThreadedRegexVM Performance seems unaffacted, but memory usage should be lowered as the Thread struct is 4 bytes instead of 16.
2018-04-27 00:18:04 +02:00
if (thread.saves >= 0)
++m_saves[thread.saves]->refcount;
Regex: Use a custom 'DualThreadStack' structure to hold thread info Instead of using two vectors, we can hold both current and next threads in a single buffer, with stacks growing on each end. Benchmarking shows this to be slightly faster, and should use less memory.
2018-04-29 08:04:26 +02:00
m_threads.push_current({thread.inst, thread.saves});
Use indices instead of pointers for saves/instruction in ThreadedRegexVM Performance seems unaffacted, but memory usage should be lowered as the Thread struct is 4 bytes instead of 16.
2018-04-27 00:18:04 +02:00
thread.inst = static_cast<uint16_t>(inst.param);
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
break;
}
case CompiledRegex::Save:
{
Regex: refactor handling of Saves slightly, do not create them until really needed
2017-10-20 10:55:38 +02:00
if (no_saves)
Regex: Add a NoSaves RegexExecFlags to disable saving positions
2017-10-03 13:07:44 +02:00
break;
Use indices instead of pointers for saves/instruction in ThreadedRegexVM Performance seems unaffacted, but memory usage should be lowered as the Thread struct is 4 bytes instead of 16.
2018-04-27 00:18:04 +02:00
if (thread.saves < 0)
Regex: refactor handling of Saves slightly, do not create them until really needed
2017-10-20 10:55:38 +02:00
thread.saves = new_saves<false>(nullptr);
Use indices instead of pointers for saves/instruction in ThreadedRegexVM Performance seems unaffacted, but memory usage should be lowered as the Thread struct is 4 bytes instead of 16.
2018-04-27 00:18:04 +02:00
else if (m_saves[thread.saves]->refcount > 1)
Regex: store saves in a copy on write structure
2017-10-03 04:54:43 +02:00
{
Use indices instead of pointers for saves/instruction in ThreadedRegexVM Performance seems unaffacted, but memory usage should be lowered as the Thread struct is 4 bytes instead of 16.
2018-04-27 00:18:04 +02:00
--m_saves[thread.saves]->refcount;
thread.saves = new_saves<true>(m_saves[thread.saves]->pos);
Regex: store saves in a copy on write structure
2017-10-03 04:54:43 +02:00
}
Use BufferCoord sentinel type for regex matching on BufferIterators BufferIterators are large-ish, and need to check the buffer pointer on comparison. Checking against a coord is just a 64 bit comparison.
2018-11-01 11:51:10 +01:00
m_saves[thread.saves]->pos[inst.param] = base(pos);
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
break;
}
Regex: Replace generic 'Matchers' with specialized functionality Introduce CharacterClass and CharacterType Regex Op, and optimize their evaluation.
2017-11-25 11:14:15 +01:00
case CompiledRegex::Class:
Use BufferCoord sentinel type for regex matching on BufferIterators BufferIterators are large-ish, and need to check the buffer pointer on comparison. Checking against a coord is just a 64 bit comparison.
2018-11-01 11:51:10 +01:00
if (base(pos) == config.end)
Regex: tag instructions as scheduled as well instead of searching And a few more code cleanup in the ThreadedRegexVM
2017-10-07 13:58:10 +02:00
return StepResult::Failed;
Regex: Replace generic 'Matchers' with specialized functionality Introduce CharacterClass and CharacterType Regex Op, and optimize their evaluation.
2017-11-25 11:14:15 +01:00
return is_character_class(m_program.character_classes[inst.param], *pos) ?
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
StepResult::Consumed : StepResult::Failed;
Regex: Replace generic 'Matchers' with specialized functionality Introduce CharacterClass and CharacterType Regex Op, and optimize their evaluation.
2017-11-25 11:14:15 +01:00
case CompiledRegex::CharacterType:
Use BufferCoord sentinel type for regex matching on BufferIterators BufferIterators are large-ish, and need to check the buffer pointer on comparison. Checking against a coord is just a 64 bit comparison.
2018-11-01 11:51:10 +01:00
if (base(pos) == config.end)
Regex: Replace generic 'Matchers' with specialized functionality Introduce CharacterClass and CharacterType Regex Op, and optimize their evaluation.
2017-11-25 11:14:15 +01:00
return StepResult::Failed;
return is_ctype((CharacterType)inst.param, *pos) ?
Cleanup some trailing whitespaces and double semicolon
2018-07-26 13:56:34 +02:00
StepResult::Consumed : StepResult::Failed;
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
case CompiledRegex::LineStart:
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
if (not is_line_start(pos, config))
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
return StepResult::Failed;
break;
case CompiledRegex::LineEnd:
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
if (not is_line_end(pos, config))
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
return StepResult::Failed;
break;
case CompiledRegex::WordBoundary:
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
if (not is_word_boundary(pos, config))
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
return StepResult::Failed;
break;
case CompiledRegex::NotWordBoundary:
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
if (is_word_boundary(pos, config))
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
return StepResult::Failed;
break;
case CompiledRegex::SubjectBegin:
Use BufferCoord sentinel type for regex matching on BufferIterators BufferIterators are large-ish, and need to check the buffer pointer on comparison. Checking against a coord is just a 64 bit comparison.
2018-11-01 11:51:10 +01:00
if (base(pos) != config.subject_begin)
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
return StepResult::Failed;
break;
case CompiledRegex::SubjectEnd:
Use BufferCoord sentinel type for regex matching on BufferIterators BufferIterators are large-ish, and need to check the buffer pointer on comparison. Checking against a coord is just a 64 bit comparison.
2018-11-01 11:51:10 +01:00
if (base(pos) != config.subject_end)
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
return StepResult::Failed;
break;
case CompiledRegex::LookAhead:
case CompiledRegex::NegativeLookAhead:
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
if (lookaround<MatchDirection::Forward, false>(inst.param, pos, config) !=
Regex: Fix support for ignore case in lookarounds
2017-10-10 05:21:21 +02:00
(inst.op == CompiledRegex::LookAhead))
return StepResult::Failed;
break;
case CompiledRegex::LookAhead_IgnoreCase:
case CompiledRegex::NegativeLookAhead_IgnoreCase:
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
if (lookaround<MatchDirection::Forward, true>(inst.param, pos, config) !=
Regex: Fix support for ignore case in lookarounds
2017-10-10 05:21:21 +02:00
(inst.op == CompiledRegex::LookAhead_IgnoreCase))
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
return StepResult::Failed;
break;
case CompiledRegex::LookBehind:
case CompiledRegex::NegativeLookBehind:
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
if (lookaround<MatchDirection::Backward, false>(inst.param, pos, config) !=
Regex: Fix support for ignore case in lookarounds
2017-10-10 05:21:21 +02:00
(inst.op == CompiledRegex::LookBehind))
return StepResult::Failed;
break;
case CompiledRegex::LookBehind_IgnoreCase:
case CompiledRegex::NegativeLookBehind_IgnoreCase:
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
if (lookaround<MatchDirection::Backward, true>(inst.param, pos, config) !=
Regex: Fix support for ignore case in lookarounds
2017-10-10 05:21:21 +02:00
(inst.op == CompiledRegex::LookBehind_IgnoreCase))
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
return StepResult::Failed;
break;
Regex: Go back to instruction based search of next start The previous method, which was a bit faster in the general use case, can hit some cases where we get quadratic behaviour and very slow matching. By using an instruction, we can guarantee our complexity of O(N*M) as we will never have more than N threads (N being the instruction count) and we run the threads once per codepoint in the subject string. That slows down the general case slightly, but ensure we dont have pathological cases. This new version is much faster than the previous instruction based search because it does not use a plain `.*` searcher, but a specific, smarter instruction specialized for finding the next start if we are in the correct conditions.
2017-10-20 09:17:02 +02:00
case CompiledRegex::FindNextStart:
Regex: Use a custom 'DualThreadStack' structure to hold thread info Instead of using two vectors, we can hold both current and next threads in a single buffer, with stacks growing on each end. Benchmarking shows this to be slightly faster, and should use less memory.
2018-04-29 08:04:26 +02:00
kak_assert(m_threads.current_is_empty()); // search thread should by construction be the lower priority one
if (m_threads.next_is_empty())
Regex: Go back to instruction based search of next start The previous method, which was a bit faster in the general use case, can hit some cases where we get quadratic behaviour and very slow matching. By using an instruction, we can guarantee our complexity of O(N*M) as we will never have more than N threads (N being the instruction count) and we run the threads once per codepoint in the subject string. That slows down the general case slightly, but ensure we dont have pathological cases. This new version is much faster than the previous instruction based search because it does not use a plain `.*` searcher, but a specific, smarter instruction specialized for finding the next start if we are in the correct conditions.
2017-10-20 09:17:02 +02:00
return StepResult::FindNextStart;
return StepResult::Consumed;
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
case CompiledRegex::Match:
return StepResult::Matched;
}
}
return StepResult::Failed;
}
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
bool exec_program(EffectiveIt pos, const ExecConfig& config)
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
{
Regex: Use a custom 'DualThreadStack' structure to hold thread info Instead of using two vectors, we can hold both current and next threads in a single buffer, with stacks growing on each end. Benchmarking shows this to be slightly faster, and should use less memory.
2018-04-29 08:04:26 +02:00
kak_assert(m_threads.current_is_empty() and m_threads.next_is_empty());
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
release_saves(m_captures);
Use indices instead of pointers for saves/instruction in ThreadedRegexVM Performance seems unaffacted, but memory usage should be lowered as the Thread struct is 4 bytes instead of 16.
2018-04-27 00:18:04 +02:00
m_captures = -1;
Regex: Use a custom 'DualThreadStack' structure to hold thread info Instead of using two vectors, we can hold both current and next threads in a single buffer, with stacks growing on each end. Benchmarking shows this to be slightly faster, and should use less memory.
2018-04-29 08:04:26 +02:00
m_threads.push_current({static_cast<int16_t>(&config.instructions[0] - &m_program.instructions[0]), -1});
Regex: Support forward and backward matching code in the same CompiledRegex No need to have two separate regexes to handle forward and backward matching, just passing RegexCompileFlags::Backward will add support for backward matching to the regex. For backward only regex, pass RegexCompileFlags::NoForward as well to disable generation of forward matching code.
2017-12-01 12:57:02 +01:00
const auto& start_desc = direction == MatchDirection::Forward ? m_program.forward_start_desc
: m_program.backward_start_desc;
Regex: Add a NoSaves RegexExecFlags to disable saving positions
2017-10-03 13:07:44 +02:00
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
uint16_t current_step = -1;
Regex: refactor ThreadedRegexVM::exec_from code
2017-10-07 13:27:06 +02:00
bool found_match = false;
Regex: tag instructions as scheduled as well instead of searching And a few more code cleanup in the ThreadedRegexVM
2017-10-07 13:58:10 +02:00
while (true) // Iterate on all codepoints and once at the end
Regex: refactor ThreadedRegexVM::exec_from code
2017-10-07 13:27:06 +02:00
{
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
if (++current_step == 0)
Regex: remove the need to a processed inst vector Identify each step with a counter, and check if the instruction was already processed this step. This makes the matching faster, by removing the need to maintain a vector of instructions executed this step.
2017-10-14 06:58:42 +02:00
{
// We wrapped, avoid potential collision on inst.last_step by resetting them
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
for (auto& inst : config.instructions)
Regex: remove the need to a processed inst vector Identify each step with a counter, and check if the instruction was already processed this step. This makes the matching faster, by removing the need to maintain a vector of instructions executed this step.
2017-10-14 06:58:42 +02:00
inst.last_step = 0;
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
current_step = 1; // step 0 is never valid
Regex: remove the need to a processed inst vector Identify each step with a counter, and check if the instruction was already processed this step. This makes the matching faster, by removing the need to maintain a vector of instructions executed this step.
2017-10-14 06:58:42 +02:00
}
Regex: Go back to instruction based search of next start The previous method, which was a bit faster in the general use case, can hit some cases where we get quadratic behaviour and very slow matching. By using an instruction, we can guarantee our complexity of O(N*M) as we will never have more than N threads (N being the instruction count) and we run the threads once per codepoint in the subject string. That slows down the general case slightly, but ensure we dont have pathological cases. This new version is much faster than the previous instruction based search because it does not use a plain `.*` searcher, but a specific, smarter instruction specialized for finding the next start if we are in the correct conditions.
2017-10-20 09:17:02 +02:00
bool find_next_start = false;
Regex: Use a custom 'DualThreadStack' structure to hold thread info Instead of using two vectors, we can hold both current and next threads in a single buffer, with stacks growing on each end. Benchmarking shows this to be slightly faster, and should use less memory.
2018-04-29 08:04:26 +02:00
while (not m_threads.current_is_empty())
Regex: tag instructions as scheduled as well instead of searching And a few more code cleanup in the ThreadedRegexVM
2017-10-07 13:58:10 +02:00
{
Regex: Use a custom 'DualThreadStack' structure to hold thread info Instead of using two vectors, we can hold both current and next threads in a single buffer, with stacks growing on each end. Benchmarking shows this to be slightly faster, and should use less memory.
2018-04-29 08:04:26 +02:00
auto thread = m_threads.pop_current();
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
switch (step(pos, current_step, thread, config))
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
{
Regex: Refactor thread handling in ThreadedRegexVM
2017-10-03 12:00:52 +02:00
case StepResult::Matched:
Use BufferCoord sentinel type for regex matching on BufferIterators BufferIterators are large-ish, and need to check the buffer pointer on comparison. Checking against a coord is just a 64 bit comparison.
2018-11-01 11:51:10 +01:00
if ((base(pos) != config.end and not (config.flags & RegexExecFlags::Search)) or
(config.flags & RegexExecFlags::NotInitialNull and base(pos) == config.begin))
Regex: Ensure we dont have a thread explosion in ThreadedRegexVM Always remove threads with lower priority that end up on the same instruction as a higher priority thread (as we know they will behave the same from now on)
2017-10-02 10:24:38 +02:00
{
Regex: Refactor thread handling in ThreadedRegexVM
2017-10-03 12:00:52 +02:00
release_saves(thread.saves);
Regex: introduce RegexExecFlags to control various behaviours
2017-10-02 16:34:57 +02:00
continue;
Regex: Ensure we dont have a thread explosion in ThreadedRegexVM Always remove threads with lower priority that end up on the same instruction as a higher priority thread (as we know they will behave the same from now on)
2017-10-02 10:24:38 +02:00
}
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
Regex: More code tweaks and cleanups in ThreadedRegexVM
2017-10-04 14:11:15 +02:00
release_saves(m_captures);
Regex: small code tweak
2017-10-04 05:28:58 +02:00
m_captures = thread.saves;
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
found_match = true;
Regex: refactor handling of Saves slightly, do not create them until really needed
2017-10-20 10:55:38 +02:00
// remove this and lower priority threads
Regex: Use a custom 'DualThreadStack' structure to hold thread info Instead of using two vectors, we can hold both current and next threads in a single buffer, with stacks growing on each end. Benchmarking shows this to be slightly faster, and should use less memory.
2018-04-29 08:04:26 +02:00
while (not m_threads.current_is_empty())
release_saves(m_threads.pop_current().saves);
Regex: Refactor thread handling in ThreadedRegexVM
2017-10-03 12:00:52 +02:00
break;
case StepResult::Failed:
release_saves(thread.saves);
break;
case StepResult::Consumed:
Use indices instead of pointers for saves/instruction in ThreadedRegexVM Performance seems unaffacted, but memory usage should be lowered as the Thread struct is 4 bytes instead of 16.
2018-04-27 00:18:04 +02:00
if (m_program.instructions[thread.inst].scheduled)
Regex: tag instructions as scheduled as well instead of searching And a few more code cleanup in the ThreadedRegexVM
2017-10-07 13:58:10 +02:00
{
Regex: Refactor thread handling in ThreadedRegexVM
2017-10-03 12:00:52 +02:00
release_saves(thread.saves);
Regex: tag instructions as scheduled as well instead of searching And a few more code cleanup in the ThreadedRegexVM
2017-10-07 13:58:10 +02:00
continue;
}
Use indices instead of pointers for saves/instruction in ThreadedRegexVM Performance seems unaffacted, but memory usage should be lowered as the Thread struct is 4 bytes instead of 16.
2018-04-27 00:18:04 +02:00
m_program.instructions[thread.inst].scheduled = true;
Regex: Use a custom 'DualThreadStack' structure to hold thread info Instead of using two vectors, we can hold both current and next threads in a single buffer, with stacks growing on each end. Benchmarking shows this to be slightly faster, and should use less memory.
2018-04-29 08:04:26 +02:00
m_threads.push_next(thread);
Regex: Refactor thread handling in ThreadedRegexVM
2017-10-03 12:00:52 +02:00
break;
Regex: Go back to instruction based search of next start The previous method, which was a bit faster in the general use case, can hit some cases where we get quadratic behaviour and very slow matching. By using an instruction, we can guarantee our complexity of O(N*M) as we will never have more than N threads (N being the instruction count) and we run the threads once per codepoint in the subject string. That slows down the general case slightly, but ensure we dont have pathological cases. This new version is much faster than the previous instruction based search because it does not use a plain `.*` searcher, but a specific, smarter instruction specialized for finding the next start if we are in the correct conditions.
2017-10-20 09:17:02 +02:00
case StepResult::FindNextStart:
Regex: Use a custom 'DualThreadStack' structure to hold thread info Instead of using two vectors, we can hold both current and next threads in a single buffer, with stacks growing on each end. Benchmarking shows this to be slightly faster, and should use less memory.
2018-04-29 08:04:26 +02:00
m_threads.push_next(thread);
Regex: Go back to instruction based search of next start The previous method, which was a bit faster in the general use case, can hit some cases where we get quadratic behaviour and very slow matching. By using an instruction, we can guarantee our complexity of O(N*M) as we will never have more than N threads (N being the instruction count) and we run the threads once per codepoint in the subject string. That slows down the general case slightly, but ensure we dont have pathological cases. This new version is much faster than the previous instruction based search because it does not use a plain `.*` searcher, but a specific, smarter instruction specialized for finding the next start if we are in the correct conditions.
2017-10-20 09:17:02 +02:00
find_next_start = true;
break;
Regex: Ensure we dont have a thread explosion in ThreadedRegexVM Always remove threads with lower priority that end up on the same instruction as a higher priority thread (as we know they will behave the same from now on)
2017-10-02 10:24:38 +02:00
}
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
}
Regex: Use a custom 'DualThreadStack' structure to hold thread info Instead of using two vectors, we can hold both current and next threads in a single buffer, with stacks growing on each end. Benchmarking shows this to be slightly faster, and should use less memory.
2018-04-29 08:04:26 +02:00
for (auto& thread : m_threads.next_threads())
Use indices instead of pointers for saves/instruction in ThreadedRegexVM Performance seems unaffacted, but memory usage should be lowered as the Thread struct is 4 bytes instead of 16.
2018-04-27 00:18:04 +02:00
m_program.instructions[thread.inst].scheduled = false;
Regex: Only reset processed and scheduled flags on relevant instructions On big regex, reseting all those flags on all instructions for each character can become the dominant operation. Track that actual instructions index processed (the scheduled are already tracked in the next_threads vector), and only reset these.
2017-10-11 04:24:05 +02:00
Use BufferCoord sentinel type for regex matching on BufferIterators BufferIterators are large-ish, and need to check the buffer pointer on comparison. Checking against a coord is just a 64 bit comparison.
2018-11-01 11:51:10 +01:00
if (base(pos) == config.end or m_threads.next_is_empty() or
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
(found_match and (config.flags & RegexExecFlags::AnyMatch)))
Regex: refactor handling of Saves slightly, do not create them until really needed
2017-10-20 10:55:38 +02:00
{
Regex: Use a custom 'DualThreadStack' structure to hold thread info Instead of using two vectors, we can hold both current and next threads in a single buffer, with stacks growing on each end. Benchmarking shows this to be slightly faster, and should use less memory.
2018-04-29 08:04:26 +02:00
for (auto& t : m_threads.next_threads())
Regex: refactor handling of Saves slightly, do not create them until really needed
2017-10-20 10:55:38 +02:00
release_saves(t.saves);
Regex: Use a custom 'DualThreadStack' structure to hold thread info Instead of using two vectors, we can hold both current and next threads in a single buffer, with stacks growing on each end. Benchmarking shows this to be slightly faster, and should use less memory.
2018-04-29 08:04:26 +02:00
m_threads.clear_next();
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
return found_match;
Regex: refactor handling of Saves slightly, do not create them until really needed
2017-10-20 10:55:38 +02:00
}
Regex: Refactor thread handling in ThreadedRegexVM
2017-10-03 12:00:52 +02:00
Regex: Use a custom 'DualThreadStack' structure to hold thread info Instead of using two vectors, we can hold both current and next threads in a single buffer, with stacks growing on each end. Benchmarking shows this to be slightly faster, and should use less memory.
2018-04-29 08:04:26 +02:00
m_threads.swap_next();
Regex: refactor ThreadedRegexVM::exec_from code
2017-10-07 13:27:06 +02:00
++pos;
Regex: Go back to instruction based search of next start The previous method, which was a bit faster in the general use case, can hit some cases where we get quadratic behaviour and very slow matching. By using an instruction, we can guarantee our complexity of O(N*M) as we will never have more than N threads (N being the instruction count) and we run the threads once per codepoint in the subject string. That slows down the general case slightly, but ensure we dont have pathological cases. This new version is much faster than the previous instruction based search because it does not use a plain `.*` searcher, but a specific, smarter instruction specialized for finding the next start if we are in the correct conditions.
2017-10-20 09:17:02 +02:00
Regex: Support forward and backward matching code in the same CompiledRegex No need to have two separate regexes to handle forward and backward matching, just passing RegexCompileFlags::Backward will add support for backward matching to the regex. For backward only regex, pass RegexCompileFlags::NoForward as well to disable generation of forward matching code.
2017-12-01 12:57:02 +01:00
if (find_next_start and start_desc)
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
to_next_start(pos, config.end, *start_desc);
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
}
}
Use BufferCoord sentinel type for regex matching on BufferIterators BufferIterators are large-ish, and need to check the buffer pointer on comparison. Checking against a coord is just a 64 bit comparison.
2018-11-01 11:51:10 +01:00
void to_next_start(EffectiveIt& start, const Sentinel& end, const StartDesc& start_desc)
Regex: Find potential start position using a map of valid start chars With this optimization we get close to performance parity with boost regex on the common use cases in Kakoune.
2017-10-06 07:40:27 +02:00
{
Use BufferCoord sentinel type for regex matching on BufferIterators BufferIterators are large-ish, and need to check the buffer pointer on comparison. Checking against a coord is just a 64 bit comparison.
2018-11-01 11:51:10 +01:00
while (base(start) != end)
Improve regex vm to next start performance by avoiding iterator copies
2018-10-31 11:13:14 +01:00
{
const Codepoint cp = read(start);
if (start_desc.map[(cp >= 0 and cp < StartDesc::count) ? cp : StartDesc::other])
{
--start;
return;
}
}
Regex: Find potential start position using a map of valid start chars With this optimization we get close to performance parity with boost regex on the common use cases in Kakoune.
2017-10-06 07:40:27 +02:00
}
Regex: Fix support for ignore case in lookarounds
2017-10-10 05:21:21 +02:00
template<MatchDirection look_direction, bool ignore_case>
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
bool lookaround(uint32_t index, EffectiveIt pos, const ExecConfig& config) const
Regex: Support any char and character classes in lookarounds Lookarounds still need to be fixed size, but accept character classes as well as plain literals.
2017-10-09 05:12:42 +02:00
{
Cleanup regex lookarounds implementation and reject incompatible regex Fixes #2487
2018-10-10 13:35:38 +02:00
using Lookaround = CompiledRegex::Lookaround;
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
const auto end = (look_direction == MatchDirection::Forward ? config.subject_end : config.subject_begin);
Cleanup regex lookarounds implementation and reject incompatible regex Fixes #2487
2018-10-10 13:35:38 +02:00
for (auto it = m_program.lookarounds.begin() + index; *it != Lookaround::EndOfLookaround; ++it)
Regex: Support any char and character classes in lookarounds Lookarounds still need to be fixed size, but accept character classes as well as plain literals.
2017-10-09 05:12:42 +02:00
{
Use BufferCoord sentinel type for regex matching on BufferIterators BufferIterators are large-ish, and need to check the buffer pointer on comparison. Checking against a coord is just a 64 bit comparison.
2018-11-01 11:51:10 +01:00
if (base(pos) == end)
Regex: Support any char and character classes in lookarounds Lookarounds still need to be fixed size, but accept character classes as well as plain literals.
2017-10-09 05:12:42 +02:00
return false;
Regex: slight readability improvement and workaround a potential gcc bug
2017-10-23 11:00:42 +02:00
Codepoint cp = (look_direction == MatchDirection::Forward ? *pos : *(pos-1));
Regex: Fix support for ignore case in lookarounds
2017-10-10 05:21:21 +02:00
if (ignore_case)
cp = to_lower(cp);
Cleanup regex lookarounds implementation and reject incompatible regex Fixes #2487
2018-10-10 13:35:38 +02:00
const Lookaround op = *it;
if (op == Lookaround::AnyChar)
Regex: Support any char and character classes in lookarounds Lookarounds still need to be fixed size, but accept character classes as well as plain literals.
2017-10-09 05:12:42 +02:00
{} // any character matches
Cleanup regex lookarounds implementation and reject incompatible regex Fixes #2487
2018-10-10 13:35:38 +02:00
else if (op == Lookaround::AnyCharExceptNewLine)
Use a dedicated vm op for dot when match-newline is false
2018-06-24 12:13:35 +02:00
{
if (cp == '\n')
return false;
}
Cleanup regex lookarounds implementation and reject incompatible regex Fixes #2487
2018-10-10 13:35:38 +02:00
else if (op >= Lookaround::CharacterClass and op < Lookaround::CharacterType)
Regex: Replace generic 'Matchers' with specialized functionality Introduce CharacterClass and CharacterType Regex Op, and optimize their evaluation.
2017-11-25 11:14:15 +01:00
{
Cleanup regex lookarounds implementation and reject incompatible regex Fixes #2487
2018-10-10 13:35:38 +02:00
auto index = to_underlying(op) - to_underlying(Lookaround::CharacterClass);
if (not is_character_class(m_program.character_classes[index], cp))
Regex: Replace generic 'Matchers' with specialized functionality Introduce CharacterClass and CharacterType Regex Op, and optimize their evaluation.
2017-11-25 11:14:15 +01:00
return false;
}
Cleanup regex lookarounds implementation and reject incompatible regex Fixes #2487
2018-10-10 13:35:38 +02:00
else if (op >= Lookaround::CharacterType and op < Lookaround::OpEnd)
Regex: Fix lookaround use in moon.kak (?=[A-Z]\w*) is strictly the same as (?=[A-Z]) as \w* will always at least match an empty string.
2017-10-09 05:20:05 +02:00
{
Cleanup regex lookarounds implementation and reject incompatible regex Fixes #2487
2018-10-10 13:35:38 +02:00
auto ctype = static_cast<CharacterType>(to_underlying(op) & 0xFF);
if (not is_ctype(ctype, cp))
Regex: Fix lookaround use in moon.kak (?=[A-Z]\w*) is strictly the same as (?=[A-Z]) as \w* will always at least match an empty string.
2017-10-09 05:20:05 +02:00
return false;
}
Cleanup regex lookarounds implementation and reject incompatible regex Fixes #2487
2018-10-10 13:35:38 +02:00
else if (static_cast<Codepoint>(op) != cp)
Regex: Support any char and character classes in lookarounds Lookarounds still need to be fixed size, but accept character classes as well as plain literals.
2017-10-09 05:12:42 +02:00
return false;
(look_direction == MatchDirection::Forward) ? ++pos : --pos;
}
return true;
}
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
static bool is_line_start(const EffectiveIt& pos, const ExecConfig& config)
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
{
Use BufferCoord sentinel type for regex matching on BufferIterators BufferIterators are large-ish, and need to check the buffer pointer on comparison. Checking against a coord is just a 64 bit comparison.
2018-11-01 11:51:10 +01:00
if (base(pos) == config.subject_begin)
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
return not (config.flags & RegexExecFlags::NotBeginOfLine);
Regex: Introduce RegexExecFlags::PrevAvailable Rework assertion code as well.
2017-10-08 03:22:24 +02:00
return *(pos-1) == '\n';
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
}
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
static bool is_line_end(const EffectiveIt& pos, const ExecConfig& config)
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
{
Use BufferCoord sentinel type for regex matching on BufferIterators BufferIterators are large-ish, and need to check the buffer pointer on comparison. Checking against a coord is just a 64 bit comparison.
2018-11-01 11:51:10 +01:00
if (base(pos) == config.subject_end)
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
return not (config.flags & RegexExecFlags::NotEndOfLine);
Regex: Introduce RegexExecFlags::PrevAvailable Rework assertion code as well.
2017-10-08 03:22:24 +02:00
return *pos == '\n';
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
}
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
static bool is_word_boundary(const EffectiveIt& pos, const ExecConfig& config)
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
{
Use BufferCoord sentinel type for regex matching on BufferIterators BufferIterators are large-ish, and need to check the buffer pointer on comparison. Checking against a coord is just a 64 bit comparison.
2018-11-01 11:51:10 +01:00
if (base(pos) == config.subject_begin)
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
return not (config.flags & RegexExecFlags::NotBeginOfWord);
Use BufferCoord sentinel type for regex matching on BufferIterators BufferIterators are large-ish, and need to check the buffer pointer on comparison. Checking against a coord is just a 64 bit comparison.
2018-11-01 11:51:10 +01:00
if (base(pos) == config.subject_end)
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
return not (config.flags & RegexExecFlags::NotEndOfWord);
Regex: Introduce RegexExecFlags::PrevAvailable Rework assertion code as well.
2017-10-08 03:22:24 +02:00
return is_word(*(pos-1)) != is_word(*pos);
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
}
Improve regex vm to next start performance by avoiding iterator copies
2018-10-31 11:13:14 +01:00
static Codepoint read(Utf8It& it) { return it.read(); }
static Codepoint read(std::reverse_iterator<Utf8It>& it) { Codepoint cp = *it; ++it; return cp; }
Use BufferCoord sentinel type for regex matching on BufferIterators BufferIterators are large-ish, and need to check the buffer pointer on comparison. Checking against a coord is just a 64 bit comparison.
2018-11-01 11:51:10 +01:00
static const Iterator& base(const Utf8It& it) { return it.base(); }
static Iterator base(const std::reverse_iterator<Utf8It>& it) { return it.base().base(); }
Regex: Add support for backward matching Regex can be compiled for backward matching instead of forward matching and the ThreadedRegexVM is able to iterate in reverse on the subject string to find the last match instead of the first.
2017-10-07 06:46:27 +02:00
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
const CompiledRegex& m_program;
Regex: Use a custom 'DualThreadStack' structure to hold thread info Instead of using two vectors, we can hold both current and next threads in a single buffer, with stacks growing on each end. Benchmarking shows this to be slightly faster, and should use less memory.
2018-04-29 08:04:26 +02:00
struct DualThreadStack
{
bool current_is_empty() const { return m_current == 0; }
bool next_is_empty() const { return m_next == m_capacity; }
void push_current(Thread thread) { grow_ifn(); m_data[m_current++] = thread; }
Thread pop_current() { kak_assert(m_current > 0); return m_data[--m_current]; }
void push_next(Thread thread) { grow_ifn(); m_data[--m_next] = thread; }
void clear_next() { m_next = m_capacity; }
Fix memory leak in regex execution
2018-07-25 12:57:11 +02:00
ConstArrayView<Thread> next_threads() const { return { m_data.get() + m_next, m_data.get() + m_capacity }; }
Regex: Use a custom 'DualThreadStack' structure to hold thread info Instead of using two vectors, we can hold both current and next threads in a single buffer, with stacks growing on each end. Benchmarking shows this to be slightly faster, and should use less memory.
2018-04-29 08:04:26 +02:00
void swap_next()
{
for (; m_next < m_capacity; m_current++, m_next++)
m_data[m_current] = m_data[m_next];
}
private:
void grow_ifn()
{
if (m_current != m_next)
return;
const auto new_capacity = m_capacity ? m_capacity * 2 : 4;
Thread* new_data = new Thread[new_capacity];
Fix memory leak in regex execution
2018-07-25 12:57:11 +02:00
Thread* data = m_data.get();
std::copy(data, data + m_current, new_data);
Regex: Use a custom 'DualThreadStack' structure to hold thread info Instead of using two vectors, we can hold both current and next threads in a single buffer, with stacks growing on each end. Benchmarking shows this to be slightly faster, and should use less memory.
2018-04-29 08:04:26 +02:00
const auto new_next = new_capacity - (m_capacity - m_next);
Fix memory leak in regex execution
2018-07-25 12:57:11 +02:00
std::copy(data + m_next, data + m_capacity, new_data + new_next);
Regex: Use a custom 'DualThreadStack' structure to hold thread info Instead of using two vectors, we can hold both current and next threads in a single buffer, with stacks growing on each end. Benchmarking shows this to be slightly faster, and should use less memory.
2018-04-29 08:04:26 +02:00
m_capacity = new_capacity;
m_next = new_next;
Fix memory leak in regex execution
2018-07-25 12:57:11 +02:00
m_data.reset(new_data);
Regex: Use a custom 'DualThreadStack' structure to hold thread info Instead of using two vectors, we can hold both current and next threads in a single buffer, with stacks growing on each end. Benchmarking shows this to be slightly faster, and should use less memory.
2018-04-29 08:04:26 +02:00
}
Fix memory leak in regex execution
2018-07-25 12:57:11 +02:00
std::unique_ptr<Thread[]> m_data = nullptr;
Fix integer overflow leading to bad memory access in regex execution Fixes #2481 Fixes #2480
2018-10-08 03:43:03 +02:00
int32_t m_capacity = 0; // Maximum capacity should be 2*instruction count, so 65536
int32_t m_current = 0;
int32_t m_next = 0;
Regex: Use a custom 'DualThreadStack' structure to hold thread info Instead of using two vectors, we can hold both current and next threads in a single buffer, with stacks growing on each end. Benchmarking shows this to be slightly faster, and should use less memory.
2018-04-29 08:04:26 +02:00
};
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
Regex: Use a custom 'DualThreadStack' structure to hold thread info Instead of using two vectors, we can hold both current and next threads in a single buffer, with stacks growing on each end. Benchmarking shows this to be slightly faster, and should use less memory.
2018-04-29 08:04:26 +02:00
DualThreadStack m_threads;
Regex: Introduce a Regex memory domain to track usage separately
2017-10-15 03:23:57 +02:00
Vector<Saves*, MemoryDomain::Regex> m_saves;
Use indices instead of pointers for saves/instruction in ThreadedRegexVM Performance seems unaffacted, but memory usage should be lowered as the Thread struct is 4 bytes instead of 16.
2018-04-27 00:18:04 +02:00
int16_t m_first_free = -1;
int16_t m_captures = -1;
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
};
Regex: whenever Kakoune compiles a regex, pass it to the custom impl as well That way we can see which features are missing.
2017-09-26 09:44:30 +02:00
}
Regex: Add initial, exploratory work on a custom regex engine
2017-09-17 10:50:53 +02:00
#endif // regex_impl_hh_INCLUDED
Reference in New Issue Copy Permalink