xenia/home
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
083bf82c23
home/src/regex_impl.hh

675 lines
23 KiB
C++
Raw Normal View History

Regex: Add initial, exploratory work on a custom regex engine
2017-09-17 10:50:53 +02:00
#ifndef regex_impl_hh_INCLUDED
#define regex_impl_hh_INCLUDED
Regex: Add support for backward matching Regex can be compiled for backward matching instead of forward matching and the ThreadedRegexVM is able to iterate in reverse on the subject string to find the last match instead of the first.
2017-10-07 06:46:27 +02:00
#include "exception.hh"
Regex: remove use of buffer_utils.hh from regex_impl.cc
2017-10-07 04:43:21 +02:00
#include "flags.hh"
#include "ref_ptr.hh"
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
#include "unicode.hh"
#include "utf8.hh"
#include "vector.hh"
Cleanup regex lookarounds implementation and reject incompatible regex Fixes #2487
2018-10-10 13:35:38 +02:00
#include "utils.hh"
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
Regex: whenever Kakoune compiles a regex, pass it to the custom impl as well That way we can see which features are missing.
2017-09-26 09:44:30 +02:00
namespace Kakoune
{
Regex: switch to custom impl, use boost for checking
2017-10-09 08:04:14 +02:00
struct regex_error : runtime_error
{
using runtime_error::runtime_error;
};
Regex: Replace generic 'Matchers' with specialized functionality Introduce CharacterClass and CharacterType Regex Op, and optimize their evaluation.
2017-11-25 11:14:15 +01:00
enum class CharacterType : unsigned char
{
None = 0,
Regex: Tweak is_ctype implementation style
2017-11-27 17:13:42 +01:00
Whitespace = 1 << 0,
HorizontalWhitespace = 1 << 1,
Word = 1 << 2,
Regex: Replace generic 'Matchers' with specialized functionality Introduce CharacterClass and CharacterType Regex Op, and optimize their evaluation.
2017-11-25 11:14:15 +01:00
Digit = 1 << 3,
Regex: Tweak is_ctype implementation style
2017-11-27 17:13:42 +01:00
NotWhitespace = 1 << 4,
NotHorizontalWhitespace = 1 << 5,
NotWord = 1 << 6,
Regex: Replace generic 'Matchers' with specialized functionality Introduce CharacterClass and CharacterType Regex Op, and optimize their evaluation.
2017-11-25 11:14:15 +01:00
NotDigit = 1 << 7
};
constexpr bool with_bit_ops(Meta::Type<CharacterType>) { return true; }
struct CharacterClass
{
struct Range { Codepoint min, max; };
Vector<Range, MemoryDomain::Regex> ranges;
CharacterType ctypes = CharacterType::None;
bool negative = false;
bool ignore_case = false;
};
bool is_character_class(const CharacterClass& character_class, Codepoint cp);
bool is_ctype(CharacterType ctype, Codepoint cp);
Regex: Introduce a Regex memory domain to track usage separately
2017-10-15 03:23:57 +02:00
struct CompiledRegex : RefCountable, UseMemoryDomain<MemoryDomain::Regex>
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
{
enum Op : char
{
Match,
Literal,
AnyChar,
Use a dedicated vm op for dot when match-newline is false
2018-06-24 12:13:35 +02:00
AnyCharExceptNewLine,
Reduce the amount of Regex VM Instruction code Merge all lookarounds into the same instruction, merge splits, merge literal ignore case with literal... Besides reducing the amount of almost duplicated code, this improves performance by reducing pressure on the (often failing) branch target prediction for instruction dispatching by moving branches into the instruction code themselves where they are more likely to be well predicted.
2020-07-25 06:07:57 +02:00
CharClass,
CharType,
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
Jump,
Reduce the amount of Regex VM Instruction code Merge all lookarounds into the same instruction, merge splits, merge literal ignore case with literal... Besides reducing the amount of almost duplicated code, this improves performance by reducing pressure on the (often failing) branch target prediction for instruction dispatching by moving branches into the instruction code themselves where they are more likely to be well predicted.
2020-07-25 06:07:57 +02:00
Split,
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
Save,
Reduce the amount of Regex VM Instruction code Merge all lookarounds into the same instruction, merge splits, merge literal ignore case with literal... Besides reducing the amount of almost duplicated code, this improves performance by reducing pressure on the (often failing) branch target prediction for instruction dispatching by moving branches into the instruction code themselves where they are more likely to be well predicted.
2020-07-25 06:07:57 +02:00
LineAssertion,
SubjectAssertion,
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
WordBoundary,
Reduce the amount of Regex VM Instruction code Merge all lookarounds into the same instruction, merge splits, merge literal ignore case with literal... Besides reducing the amount of almost duplicated code, this improves performance by reducing pressure on the (often failing) branch target prediction for instruction dispatching by moving branches into the instruction code themselves where they are more likely to be well predicted.
2020-07-25 06:07:57 +02:00
LookAround,
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
};
Cleanup regex lookarounds implementation and reject incompatible regex Fixes #2487
2018-10-10 13:35:38 +02:00
enum class Lookaround : Codepoint
{
OpBegin = 0xF0000,
AnyChar = 0xF0000,
AnyCharExceptNewLine = 0xF0001,
CharacterClass = 0xF0002,
CharacterType = 0xF8000,
OpEnd = 0xFFFFF,
EndOfLookaround = static_cast<Codepoint>(-1)
};
Reduce the amount of Regex VM Instruction code Merge all lookarounds into the same instruction, merge splits, merge literal ignore case with literal... Besides reducing the amount of almost duplicated code, this improves performance by reducing pressure on the (often failing) branch target prediction for instruction dispatching by moving branches into the instruction code themselves where they are more likely to be well predicted.
2020-07-25 06:07:57 +02:00
union Param
{
struct Literal
{
uint32_t codepoint : 24;
bool ignore_case : 1;
} literal;
int16_t character_class_index;
CharacterType character_type;
int16_t jump_target;
int16_t save_index;
struct Split
{
int16_t target;
bool prioritize_parent : 1;
} split;
bool line_start;
bool subject_begin;
bool word_boundary_positive;
struct Lookaround
{
int16_t index;
bool ahead : 1;
bool positive : 1;
bool ignore_case : 1;
} lookaround;
};
static_assert(sizeof(Param) == 4);
Regex: abandon bytecode and just use a simple list of instructions Makes the code simpler.
2017-10-07 12:51:32 +02:00
struct Instruction
{
Op op;
Regex: comment the mutables in CompiledRegex::Instruction and fix their init
2017-10-15 03:34:49 +02:00
// Those mutables are used during execution
Regex: tag instructions as scheduled as well instead of searching And a few more code cleanup in the ThreadedRegexVM
2017-10-07 13:58:10 +02:00
mutable bool scheduled;
Regex: remove the need to a processed inst vector Identify each step with a counter, and check if the instruction was already processed this step. This makes the matching faster, by removing the need to maintain a vector of instructions executed this step.
2017-10-14 06:58:42 +02:00
mutable uint16_t last_step;
Reduce the amount of Regex VM Instruction code Merge all lookarounds into the same instruction, merge splits, merge literal ignore case with literal... Besides reducing the amount of almost duplicated code, this improves performance by reducing pressure on the (often failing) branch target prediction for instruction dispatching by moving branches into the instruction code themselves where they are more likely to be well predicted.
2020-07-25 06:07:57 +02:00
Param param;
Regex: abandon bytecode and just use a simple list of instructions Makes the code simpler.
2017-10-07 12:51:32 +02:00
};
Reduce the amount of Regex VM Instruction code Merge all lookarounds into the same instruction, merge splits, merge literal ignore case with literal... Besides reducing the amount of almost duplicated code, this improves performance by reducing pressure on the (often failing) branch target prediction for instruction dispatching by moving branches into the instruction code themselves where they are more likely to be well predicted.
2020-07-25 06:07:57 +02:00
static_assert(sizeof(Instruction) == 8);
Regex: abandon bytecode and just use a simple list of instructions Makes the code simpler.
2017-10-07 12:51:32 +02:00
explicit operator bool() const { return not instructions.empty(); }
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
Add support for named captures to the regex impl and regex highlighter ECMAScript is adding support for it, and it is a pretty isolated change to do. Fixes #2293
2019-01-03 12:52:15 +01:00
struct NamedCapture
{
String name;
uint32_t index;
};
Regex: Introduce a Regex memory domain to track usage separately
2017-10-15 03:23:57 +02:00
Vector<Instruction, MemoryDomain::Regex> instructions;
Regex: Replace generic 'Matchers' with specialized functionality Introduce CharacterClass and CharacterType Regex Op, and optimize their evaluation.
2017-11-25 11:14:15 +01:00
Vector<CharacterClass, MemoryDomain::Regex> character_classes;
Cleanup regex lookarounds implementation and reject incompatible regex Fixes #2487
2018-10-10 13:35:38 +02:00
Vector<Lookaround, MemoryDomain::Regex> lookarounds;
Add support for named captures to the regex impl and regex highlighter ECMAScript is adding support for it, and it is a pretty isolated change to do. Fixes #2293
2019-01-03 12:52:15 +01:00
Vector<NamedCapture, MemoryDomain::Regex> named_captures;
Regex: Support forward and backward matching code in the same CompiledRegex No need to have two separate regexes to handle forward and backward matching, just passing RegexCompileFlags::Backward will add support for backward matching to the regex. For backward only regex, pass RegexCompileFlags::NoForward as well to disable generation of forward matching code.
2017-12-01 12:57:02 +01:00
uint32_t first_backward_inst; // -1 if no backward support, 0 if only backward, >0 if both forward and backward
uint32_t save_count;
Regex: Find potential start position using a map of valid start chars With this optimization we get close to performance parity with boost regex on the common use cases in Kakoune.
2017-10-06 07:40:27 +02:00
regex: track CompiledRegex::StartDesc in the Regex memory domain
2018-02-24 06:29:24 +01:00
struct StartDesc : UseMemoryDomain<MemoryDomain::Regex>
Regex: fix lookarounds handling when computing starting chars
2017-10-09 12:19:36 +02:00
{
Regex: Use only 128 characters in start desc and encode others as 0 Using 257 was using lots of memory for no good reason, as > 127 codepoint are not common enough to be treated specially.
2018-04-29 08:42:46 +02:00
static constexpr Codepoint count = 128;
static constexpr Codepoint other = 0;
bool map[count];
Regex: fix lookarounds handling when computing starting chars
2017-10-09 12:19:36 +02:00
};
Regex: put the other char boolean inside the general start char map
2017-10-21 04:04:08 +02:00
Regex: Support forward and backward matching code in the same CompiledRegex No need to have two separate regexes to handle forward and backward matching, just passing RegexCompileFlags::Backward will add support for backward matching to the regex. For backward only regex, pass RegexCompileFlags::NoForward as well to disable generation of forward matching code.
2017-12-01 12:57:02 +01:00
std::unique_ptr<StartDesc> forward_start_desc;
std::unique_ptr<StartDesc> backward_start_desc;
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
};
Add a debug regex command to dump regex instructions
2018-04-27 00:34:49 +02:00
String dump_regex(const CompiledRegex& program);
Regex: fix RegexCompileFlags not being an enum class
2017-10-23 11:29:03 +02:00
enum class RegexCompileFlags
Regex: switch to custom impl, use boost for checking
2017-10-09 08:04:14 +02:00
{
None = 0,
NoSubs = 1 << 0,
Regex: Support forward and backward matching code in the same CompiledRegex No need to have two separate regexes to handle forward and backward matching, just passing RegexCompileFlags::Backward will add support for backward matching to the regex. For backward only regex, pass RegexCompileFlags::NoForward as well to disable generation of forward matching code.
2017-12-01 12:57:02 +01:00
Optimize = 1 << 1,
Fix RegexCompileFlags::Backwards having the same value as Optimize That means every Optimized regex had the Backwards version compiled as well, which doubled the time it took to compile them and doubled the memory usage of regex. This should improve #2152
2018-07-19 10:34:40 +02:00
Backward = 1 << 2,
NoForward = 1 << 3,
Regex: switch to custom impl, use boost for checking
2017-10-09 08:04:14 +02:00
};
constexpr bool with_bit_ops(Meta::Type<RegexCompileFlags>) { return true; }
Regex: Support forward and backward matching code in the same CompiledRegex No need to have two separate regexes to handle forward and backward matching, just passing RegexCompileFlags::Backward will add support for backward matching to the regex. For backward only regex, pass RegexCompileFlags::NoForward as well to disable generation of forward matching code.
2017-12-01 12:57:02 +01:00
CompiledRegex compile_regex(StringView re, RegexCompileFlags flags);
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
Regex: introduce RegexExecFlags to control various behaviours
2017-10-02 16:34:57 +02:00
enum class RegexExecFlags
{
None = 0,
NotBeginOfLine = 1 << 1,
NotEndOfLine = 1 << 2,
NotBeginOfWord = 1 << 3,
NotEndOfWord = 1 << 4,
Regex: take the full subject range as a parameter To allow more general look arounds out of the actual search range, pass a second range (the actual subject). This allows us to remove various flags such as PrevAvailable or NotBeginOfSubject, which are now easy to check from the subject range. Fixes #1902
2018-03-04 19:48:10 +01:00
NotInitialNull = 1 << 5,
Regex: introduce RegexExecFlags to control various behaviours
2017-10-02 16:34:57 +02:00
};
constexpr bool with_bit_ops(Meta::Type<RegexExecFlags>) { return true; }
Split compile time regex flags from runtime ones
2018-12-27 01:29:33 +01:00
enum class RegexMode
{
Forward = 1 << 0,
Backward = 1 << 1,
Search = 1 << 2,
AnyMatch = 1 << 3,
NoSaves = 1 << 4,
};
constexpr bool with_bit_ops(Meta::Type<RegexMode>) { return true; }
constexpr bool has_direction(RegexMode mode)
{
return (bool)(mode & RegexMode::Forward) xor
(bool)(mode & RegexMode::Backward);
}
constexpr bool is_direction(RegexMode mode)
{
return has_direction(mode) and
(mode & ~(RegexMode::Forward | RegexMode::Backward)) == RegexMode{0};
}
Use BufferCoord sentinel type for regex matching on BufferIterators BufferIterators are large-ish, and need to check the buffer pointer on comparison. Checking against a coord is just a 64 bit comparison.
2018-11-01 11:51:10 +01:00
template<typename It, typename=void>
struct SentinelType { using Type = It; };
template<typename It>
struct SentinelType<It, void_t<typename It::Sentinel>> { using Type = typename It::Sentinel; };
Split compile time regex flags from runtime ones
2018-12-27 01:29:33 +01:00
template<typename Iterator, RegexMode mode>
Regex: Make ThreadedRegexVM a proper class, define a proper interface
2017-10-06 13:30:46 +02:00
class ThreadedRegexVM
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
{
Regex: Make ThreadedRegexVM a proper class, define a proper interface
2017-10-06 13:30:46 +02:00
public:
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
ThreadedRegexVM(const CompiledRegex& program)
Regex: Add support for backward matching Regex can be compiled for backward matching instead of forward matching and the ThreadedRegexVM is able to iterate in reverse on the subject string to find the last match instead of the first.
2017-10-07 06:46:27 +02:00
: m_program{program}
Regex: small code cleanup
2017-10-09 15:56:48 +02:00
{
Remove use of utf8::iterator in regex execution This avoids having two copies of the subject string bounds, one in the ExecConfig and one in the utf8 iterator.
2018-11-04 22:17:50 +01:00
kak_assert((forward and program.first_backward_inst != 0) or
Split compile time regex flags from runtime ones
2018-12-27 01:29:33 +01:00
(not forward and program.first_backward_inst != -1));
Regex: small code cleanup
2017-10-09 15:56:48 +02:00
}
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
Regex: More code tweaks and cleanups in ThreadedRegexVM
2017-10-04 14:11:15 +02:00
ThreadedRegexVM(const ThreadedRegexVM&) = delete;
Regex: Make ThreadedRegexVM a proper class, define a proper interface
2017-10-06 13:30:46 +02:00
ThreadedRegexVM& operator=(const ThreadedRegexVM&) = delete;
Regex: More code tweaks and cleanups in ThreadedRegexVM
2017-10-04 14:11:15 +02:00
Regex: Use a custom allocated buffer for Saves instead of a Vector
2017-10-04 05:14:24 +02:00
~ThreadedRegexVM()
{
for (auto* saves : m_saves)
{
for (size_t i = m_program.save_count-1; i > 0; --i)
saves->pos[i].~Iterator();
saves->~Saves();
Regex: Introduce a Regex memory domain to track usage separately
2017-10-15 03:23:57 +02:00
operator delete(saves);
Regex: Use a custom allocated buffer for Saves instead of a Vector
2017-10-04 05:14:24 +02:00
}
}
Take iterators by const-ref in ThreadedRegexVM::exec
2018-12-19 09:14:42 +01:00
bool exec(const Iterator& begin, const Iterator& end,
const Iterator& subject_begin, const Iterator& subject_end,
Regex: take the full subject range as a parameter To allow more general look arounds out of the actual search range, pass a second range (the actual subject). This allows us to remove various flags such as PrevAvailable or NotBeginOfSubject, which are now easy to check from the subject range. Fixes #1902
2018-03-04 19:48:10 +01:00
RegexExecFlags flags)
Regex: Make ThreadedRegexVM a proper class, define a proper interface
2017-10-06 13:30:46 +02:00
{
Regex: Fix handling of ^ and $ in backward matching mode
2017-10-11 13:24:01 +02:00
if (flags & RegexExecFlags::NotInitialNull and begin == end)
return false;
Split compile time regex flags from runtime ones
2018-12-27 01:29:33 +01:00
constexpr bool search = (mode & RegexMode::Search);
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
ConstArrayView<CompiledRegex::Instruction> instructions{m_program.instructions};
Remove use of utf8::iterator in regex execution This avoids having two copies of the subject string bounds, one in the ExecConfig and one in the utf8 iterator.
2018-11-04 22:17:50 +01:00
if (forward)
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
instructions = instructions.subrange(0, m_program.first_backward_inst);
else
instructions = instructions.subrange(m_program.first_backward_inst);
const ExecConfig config{
Use BufferCoord sentinel type for regex matching on BufferIterators BufferIterators are large-ish, and need to check the buffer pointer on comparison. Checking against a coord is just a 64 bit comparison.
2018-11-01 11:51:10 +01:00
Sentinel{forward ? begin : end},
Sentinel{forward ? end : begin},
Remove most regex impl special casing for backwards matching
2018-11-03 03:52:40 +01:00
Sentinel{subject_begin},
Sentinel{subject_end},
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
flags,
instructions
};
Remove use of utf8::iterator in regex execution This avoids having two copies of the subject string bounds, one in the ExecConfig and one in the utf8 iterator.
2018-11-04 22:17:50 +01:00
Iterator start = forward ? begin : end;
Remove most regex impl special casing for backwards matching
2018-11-03 03:52:40 +01:00
if (const auto& start_desc = forward ? m_program.forward_start_desc : m_program.backward_start_desc)
Regex: use StartDesc to early out when not searching Early out as well if we do not find any potential start position.
2017-12-01 08:03:03 +01:00
{
if (search)
{
Remove use of utf8::iterator in regex execution This avoids having two copies of the subject string bounds, one in the ExecConfig and one in the utf8 iterator.
2018-11-04 22:17:50 +01:00
to_next_start(start, config, *start_desc);
Use custom code instead of reverse_iterator in Regex VM
2018-11-01 22:23:39 +01:00
if (start == config.end) // If start_desc is not null, it means we consume at least one char
Regex: use StartDesc to early out when not searching Early out as well if we do not find any potential start position.
2017-12-01 08:03:03 +01:00
return false;
}
Remove use of utf8::iterator in regex execution This avoids having two copies of the subject string bounds, one in the ExecConfig and one in the utf8 iterator.
2018-11-04 22:17:50 +01:00
else if (start != config.end)
{
Do not decode utf-8 when looking for regex next start There is no need to decode as we know any non-ascii characters will be treated as Other in the StartDesc.
2019-12-04 12:00:31 +01:00
const unsigned char c = forward ? *start : *utf8::previous(start, config.end);
Fix regex alternation execution priority The ThreadedRegexVM implementation does not execute split opcodes as expected: on split the pending thread is pushed on top of the thread stack, which means that when multiple splits are executed in a row (such as with a disjunction with 3 or more branches) the last split target gets on top of the thread stack and gets executed next (when the thread from the first split target would be the expected one) Fixing this in the ThreadedRegexVM would have a performance impact as we would not be able to use a plain stack for current threads, so the best solution at the moment is to reverse the order of splits generated by a disjunction. Fixes #4519
2022-02-02 04:51:04 +01:00
if (not start_desc->map[(c < StartDesc::count) ? c : StartDesc::other])
Remove use of utf8::iterator in regex execution This avoids having two copies of the subject string bounds, one in the ExecConfig and one in the utf8 iterator.
2018-11-04 22:17:50 +01:00
return false;
}
Regex: use StartDesc to early out when not searching Early out as well if we do not find any potential start position.
2017-12-01 08:03:03 +01:00
}
Regex: Make ThreadedRegexVM a proper class, define a proper interface
2017-10-06 13:30:46 +02:00
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
return exec_program(std::move(start), config);
Regex: Make ThreadedRegexVM a proper class, define a proper interface
2017-10-06 13:30:46 +02:00
}
ArrayView<const Iterator> captures() const
{
Use indices instead of pointers for saves/instruction in ThreadedRegexVM Performance seems unaffacted, but memory usage should be lowered as the Thread struct is 4 bytes instead of 16.
2018-04-27 00:18:04 +02:00
if (m_captures >= 0)
return { m_saves[m_captures]->pos, m_program.save_count };
Regex: Make ThreadedRegexVM a proper class, define a proper interface
2017-10-06 13:30:46 +02:00
return {};
}
private:
Regex: store saves in a copy on write structure
2017-10-03 04:54:43 +02:00
struct Saves
{
Various micro performance tweaks in ThreadedRegexVM
2018-11-05 11:54:29 +01:00
int16_t refcount;
int16_t next_free;
Regex: Use a custom allocated buffer for Saves instead of a Vector
2017-10-04 05:14:24 +02:00
Iterator pos[1];
Regex: store saves in a copy on write structure
2017-10-03 04:54:43 +02:00
};
Regex: Code cleanup and refactor for Saves handling
2017-10-04 13:49:16 +02:00
template<bool copy>
Use indices instead of pointers for saves/instruction in ThreadedRegexVM Performance seems unaffacted, but memory usage should be lowered as the Thread struct is 4 bytes instead of 16.
2018-04-27 00:18:04 +02:00
int16_t new_saves(Iterator* pos)
Regex: store saves in a copy on write structure
2017-10-03 04:54:43 +02:00
{
Regex: Code cleanup and refactor for Saves handling
2017-10-04 13:49:16 +02:00
kak_assert(not copy or pos != nullptr);
const auto count = m_program.save_count;
Use indices instead of pointers for saves/instruction in ThreadedRegexVM Performance seems unaffacted, but memory usage should be lowered as the Thread struct is 4 bytes instead of 16.
2018-04-27 00:18:04 +02:00
if (m_first_free >= 0)
Regex: store saves in a copy on write structure
2017-10-03 04:54:43 +02:00
{
Use indices instead of pointers for saves/instruction in ThreadedRegexVM Performance seems unaffacted, but memory usage should be lowered as the Thread struct is 4 bytes instead of 16.
2018-04-27 00:18:04 +02:00
const int16_t res = m_first_free;
Various micro performance tweaks in ThreadedRegexVM
2018-11-05 11:54:29 +01:00
Saves& saves = *m_saves[res];
m_first_free = saves.next_free;
kak_assert(saves.refcount == 1);
Regex: Code cleanup and refactor for Saves handling
2017-10-04 13:49:16 +02:00
if (copy)
Various micro performance tweaks in ThreadedRegexVM
2018-11-05 11:54:29 +01:00
std::copy_n(pos, count, saves.pos);
Regex: Code cleanup and refactor for Saves handling
2017-10-04 13:49:16 +02:00
else
Various micro performance tweaks in ThreadedRegexVM
2018-11-05 11:54:29 +01:00
std::fill_n(saves.pos, count, Iterator{});
Regex: Code cleanup and refactor for Saves handling
2017-10-04 13:49:16 +02:00
Regex: small refactor of ThreadedRegexVM::clone_saves
2017-10-03 12:23:31 +02:00
return res;
Regex: store saves in a copy on write structure
2017-10-03 04:54:43 +02:00
}
Regex: Introduce a Regex memory domain to track usage separately
2017-10-15 03:23:57 +02:00
void* ptr = operator new (sizeof(Saves) + (count-1) * sizeof(Iterator));
Various micro performance tweaks in ThreadedRegexVM
2018-11-05 11:54:29 +01:00
Saves* saves = new (ptr) Saves{1, 0, {copy ? pos[0] : Iterator{}}};
Regex: Code cleanup and refactor for Saves handling
2017-10-04 13:49:16 +02:00
for (size_t i = 1; i < count; ++i)
new (&saves->pos[i]) Iterator{copy ? pos[i] : Iterator{}};
m_saves.push_back(saves);
Use indices instead of pointers for saves/instruction in ThreadedRegexVM Performance seems unaffacted, but memory usage should be lowered as the Thread struct is 4 bytes instead of 16.
2018-04-27 00:18:04 +02:00
return static_cast<int16_t>(m_saves.size() - 1);
Regex: store saves in a copy on write structure
2017-10-03 04:54:43 +02:00
}
Various micro performance tweaks in ThreadedRegexVM
2018-11-05 11:54:29 +01:00
void release_saves(int16_t index)
Regex: make m_current_threads and m_next_threads local variable of exec
2017-10-04 04:49:40 +02:00
{
Various micro performance tweaks in ThreadedRegexVM
2018-11-05 11:54:29 +01:00
if (index < 0)
return;
auto& saves = *m_saves[index];
if (saves.refcount == 1)
Regex: use intrusive linked list for the free saves instead of a Vector
2017-10-13 15:58:38 +02:00
{
Various micro performance tweaks in ThreadedRegexVM
2018-11-05 11:54:29 +01:00
saves.next_free = m_first_free;
m_first_free = index;
Regex: use intrusive linked list for the free saves instead of a Vector
2017-10-13 15:58:38 +02:00
}
Various micro performance tweaks in ThreadedRegexVM
2018-11-05 11:54:29 +01:00
else
--saves.refcount;
Regex: make m_current_threads and m_next_threads local variable of exec
2017-10-04 04:49:40 +02:00
};
align ThreadedRegexVM::Thread to permit fused copy optimization Aligning makes gcc able to copy a Thread object with a single 32bit mov instruction instead of two 16bits one.
2018-11-06 10:13:09 +01:00
struct alignas(int32_t) Thread
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
{
Use indices instead of pointers for saves/instruction in ThreadedRegexVM Performance seems unaffacted, but memory usage should be lowered as the Thread struct is 4 bytes instead of 16.
2018-04-27 00:18:04 +02:00
int16_t inst;
int16_t saves;
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
};
Regex: small code style tweak
2018-04-29 09:26:49 +02:00
using StartDesc = CompiledRegex::StartDesc;
Use BufferCoord sentinel type for regex matching on BufferIterators BufferIterators are large-ish, and need to check the buffer pointer on comparison. Checking against a coord is just a 64 bit comparison.
2018-11-01 11:51:10 +01:00
using Sentinel = typename SentinelType<Iterator>::Type;
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
struct ExecConfig
Regex: Only reset processed and scheduled flags on relevant instructions On big regex, reseting all those flags on all instructions for each character can become the dominant operation. Track that actual instructions index processed (the scheduled are already tracked in the next_threads vector), and only reset these.
2017-10-11 04:24:05 +02:00
{
Use BufferCoord sentinel type for regex matching on BufferIterators BufferIterators are large-ish, and need to check the buffer pointer on comparison. Checking against a coord is just a 64 bit comparison.
2018-11-01 11:51:10 +01:00
const Sentinel begin;
const Sentinel end;
const Sentinel subject_begin;
const Sentinel subject_end;
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
const RegexExecFlags flags;
ConstArrayView<CompiledRegex::Instruction> instructions;
Regex: Only reset processed and scheduled flags on relevant instructions On big regex, reseting all those flags on all instructions for each character can become the dominant operation. Track that actual instructions index processed (the scheduled are already tracked in the next_threads vector), and only reset these.
2017-10-11 04:24:05 +02:00
};
Regex: Add some comments, remove supurious semicolons
2017-10-07 10:36:53 +02:00
// Steps a thread until it consumes the current character, matches or fail
Micro-optimize regex character class/type matching Also force-inline step_thread as function call overhead has a mesurable impact.
2020-07-30 11:51:25 +02:00
[[gnu::always_inline]]
Refactor ThreadedRegexVM::exec_program to avoid branching Moving logic into step_thread instead of returning an enum to select what to run avoids the switch logic and improves run time.
2018-11-05 09:46:53 +01:00
void step_thread(const Iterator& pos, uint16_t current_step, Thread thread, const ExecConfig& config)
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
{
Do not pass thread to failed/consumed, capture it implicitely
2018-12-19 09:16:14 +01:00
auto failed = [this, &thread]() {
Refactor ThreadedRegexVM::exec_program to avoid branching Moving logic into step_thread instead of returning an enum to select what to run avoids the switch logic and improves run time.
2018-11-05 09:46:53 +01:00
release_saves(thread.saves);
};
Do not pass thread to failed/consumed, capture it implicitely
2018-12-19 09:16:14 +01:00
auto consumed = [this, &thread]() {
Refactor ThreadedRegexVM::exec_program to avoid branching Moving logic into step_thread instead of returning an enum to select what to run avoids the switch logic and improves run time.
2018-11-05 09:46:53 +01:00
if (m_program.instructions[thread.inst].scheduled)
return release_saves(thread.saves);
m_program.instructions[thread.inst].scheduled = true;
m_threads.push_next(thread);
};
Regex: directly store instruction pointer in Thread struct
2017-11-11 08:15:13 +01:00
auto* instructions = m_program.instructions.data();
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
while (true)
{
Use indices instead of pointers for saves/instruction in ThreadedRegexVM Performance seems unaffacted, but memory usage should be lowered as the Thread struct is 4 bytes instead of 16.
2018-04-27 00:18:04 +02:00
auto& inst = instructions[thread.inst++];
Regex: Improve comments and constify some variables Reword various comments to make some tricky parts of the regex engine easier to understand.
2018-02-24 07:21:15 +01:00
// if this instruction was already executed for this step in another thread,
// then this thread is redundant and can be dropped
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
if (inst.last_step == current_step)
Do not pass thread to failed/consumed, capture it implicitely
2018-12-19 09:16:14 +01:00
return failed();
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
inst.last_step = current_step;
Regex: avoid infinite loops
2017-10-07 08:25:14 +02:00
Regex: abandon bytecode and just use a simple list of instructions Makes the code simpler.
2017-10-07 12:51:32 +02:00
switch (inst.op)
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
{
Reduce the amount of Regex VM Instruction code Merge all lookarounds into the same instruction, merge splits, merge literal ignore case with literal... Besides reducing the amount of almost duplicated code, this improves performance by reducing pressure on the (often failing) branch target prediction for instruction dispatching by moving branches into the instruction code themselves where they are more likely to be well predicted.
2020-07-25 06:07:57 +02:00
case CompiledRegex::Match:
if ((pos != config.end and not (mode & RegexMode::Search)) or
(config.flags & RegexExecFlags::NotInitialNull and pos == config.begin))
return failed();
release_saves(m_captures);
m_captures = thread.saves;
m_found_match = true;
// remove lower priority threads
while (not m_threads.current_is_empty())
release_saves(m_threads.pop_current().saves);
return;
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
case CompiledRegex::Literal:
Reduce the amount of Regex VM Instruction code Merge all lookarounds into the same instruction, merge splits, merge literal ignore case with literal... Besides reducing the amount of almost duplicated code, this improves performance by reducing pressure on the (often failing) branch target prediction for instruction dispatching by moving branches into the instruction code themselves where they are more likely to be well predicted.
2020-07-25 06:07:57 +02:00
if (pos != config.end and
inst.param.literal.codepoint == (inst.param.literal.ignore_case ? to_lower(codepoint(pos, config))
: codepoint(pos, config)))
Do not pass thread to failed/consumed, capture it implicitely
2018-12-19 09:16:14 +01:00
return consumed();
return failed();
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
case CompiledRegex::AnyChar:
Do not pass thread to failed/consumed, capture it implicitely
2018-12-19 09:16:14 +01:00
return consumed();
Use a dedicated vm op for dot when match-newline is false
2018-06-24 12:13:35 +02:00
case CompiledRegex::AnyCharExceptNewLine:
Remove use of utf8::iterator in regex execution This avoids having two copies of the subject string bounds, one in the ExecConfig and one in the utf8 iterator.
2018-11-04 22:17:50 +01:00
if (pos != config.end and codepoint(pos, config) != '\n')
Do not pass thread to failed/consumed, capture it implicitely
2018-12-19 09:16:14 +01:00
return consumed();
return failed();
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
case CompiledRegex::Jump:
Reduce the amount of Regex VM Instruction code Merge all lookarounds into the same instruction, merge splits, merge literal ignore case with literal... Besides reducing the amount of almost duplicated code, this improves performance by reducing pressure on the (often failing) branch target prediction for instruction dispatching by moving branches into the instruction code themselves where they are more likely to be well predicted.
2020-07-25 06:07:57 +02:00
thread.inst = inst.param.jump_target;
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
break;
Reduce the amount of Regex VM Instruction code Merge all lookarounds into the same instruction, merge splits, merge literal ignore case with literal... Besides reducing the amount of almost duplicated code, this improves performance by reducing pressure on the (often failing) branch target prediction for instruction dispatching by moving branches into the instruction code themselves where they are more likely to be well predicted.
2020-07-25 06:07:57 +02:00
case CompiledRegex::Split:
Use indices instead of pointers for saves/instruction in ThreadedRegexVM Performance seems unaffacted, but memory usage should be lowered as the Thread struct is 4 bytes instead of 16.
2018-04-27 00:18:04 +02:00
if (thread.saves >= 0)
++m_saves[thread.saves]->refcount;
Reduce the amount of Regex VM Instruction code Merge all lookarounds into the same instruction, merge splits, merge literal ignore case with literal... Besides reducing the amount of almost duplicated code, this improves performance by reducing pressure on the (often failing) branch target prediction for instruction dispatching by moving branches into the instruction code themselves where they are more likely to be well predicted.
2020-07-25 06:07:57 +02:00
if (inst.param.split.prioritize_parent)
m_threads.push_current({inst.param.split.target, thread.saves});
else
{
m_threads.push_current(thread);
thread.inst = inst.param.split.target;
}
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
break;
case CompiledRegex::Save:
Split compile time regex flags from runtime ones
2018-12-27 01:29:33 +01:00
if (mode & RegexMode::NoSaves)
Regex: Add a NoSaves RegexExecFlags to disable saving positions
2017-10-03 13:07:44 +02:00
break;
Use indices instead of pointers for saves/instruction in ThreadedRegexVM Performance seems unaffacted, but memory usage should be lowered as the Thread struct is 4 bytes instead of 16.
2018-04-27 00:18:04 +02:00
if (thread.saves < 0)
Regex: refactor handling of Saves slightly, do not create them until really needed
2017-10-20 10:55:38 +02:00
thread.saves = new_saves<false>(nullptr);
Use indices instead of pointers for saves/instruction in ThreadedRegexVM Performance seems unaffacted, but memory usage should be lowered as the Thread struct is 4 bytes instead of 16.
2018-04-27 00:18:04 +02:00
else if (m_saves[thread.saves]->refcount > 1)
Regex: store saves in a copy on write structure
2017-10-03 04:54:43 +02:00
{
Use indices instead of pointers for saves/instruction in ThreadedRegexVM Performance seems unaffacted, but memory usage should be lowered as the Thread struct is 4 bytes instead of 16.
2018-04-27 00:18:04 +02:00
--m_saves[thread.saves]->refcount;
thread.saves = new_saves<true>(m_saves[thread.saves]->pos);
Regex: store saves in a copy on write structure
2017-10-03 04:54:43 +02:00
}
Reduce the amount of Regex VM Instruction code Merge all lookarounds into the same instruction, merge splits, merge literal ignore case with literal... Besides reducing the amount of almost duplicated code, this improves performance by reducing pressure on the (often failing) branch target prediction for instruction dispatching by moving branches into the instruction code themselves where they are more likely to be well predicted.
2020-07-25 06:07:57 +02:00
m_saves[thread.saves]->pos[inst.param.save_index] = pos;
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
break;
Reduce the amount of Regex VM Instruction code Merge all lookarounds into the same instruction, merge splits, merge literal ignore case with literal... Besides reducing the amount of almost duplicated code, this improves performance by reducing pressure on the (often failing) branch target prediction for instruction dispatching by moving branches into the instruction code themselves where they are more likely to be well predicted.
2020-07-25 06:07:57 +02:00
case CompiledRegex::CharClass:
Use custom code instead of reverse_iterator in Regex VM
2018-11-01 22:23:39 +01:00
if (pos == config.end)
Do not pass thread to failed/consumed, capture it implicitely
2018-12-19 09:16:14 +01:00
return failed();
Reduce the amount of Regex VM Instruction code Merge all lookarounds into the same instruction, merge splits, merge literal ignore case with literal... Besides reducing the amount of almost duplicated code, this improves performance by reducing pressure on the (often failing) branch target prediction for instruction dispatching by moving branches into the instruction code themselves where they are more likely to be well predicted.
2020-07-25 06:07:57 +02:00
return is_character_class(m_program.character_classes[inst.param.character_class_index], codepoint(pos, config)) ?
Do not pass thread to failed/consumed, capture it implicitely
2018-12-19 09:16:14 +01:00
consumed() : failed();
Reduce the amount of Regex VM Instruction code Merge all lookarounds into the same instruction, merge splits, merge literal ignore case with literal... Besides reducing the amount of almost duplicated code, this improves performance by reducing pressure on the (often failing) branch target prediction for instruction dispatching by moving branches into the instruction code themselves where they are more likely to be well predicted.
2020-07-25 06:07:57 +02:00
case CompiledRegex::CharType:
Use custom code instead of reverse_iterator in Regex VM
2018-11-01 22:23:39 +01:00
if (pos == config.end)
Do not pass thread to failed/consumed, capture it implicitely
2018-12-19 09:16:14 +01:00
return failed();
Reduce the amount of Regex VM Instruction code Merge all lookarounds into the same instruction, merge splits, merge literal ignore case with literal... Besides reducing the amount of almost duplicated code, this improves performance by reducing pressure on the (often failing) branch target prediction for instruction dispatching by moving branches into the instruction code themselves where they are more likely to be well predicted.
2020-07-25 06:07:57 +02:00
return is_ctype(inst.param.character_type, codepoint(pos, config)) ?
Do not pass thread to failed/consumed, capture it implicitely
2018-12-19 09:16:14 +01:00
consumed() : failed();
Reduce the amount of Regex VM Instruction code Merge all lookarounds into the same instruction, merge splits, merge literal ignore case with literal... Besides reducing the amount of almost duplicated code, this improves performance by reducing pressure on the (often failing) branch target prediction for instruction dispatching by moving branches into the instruction code themselves where they are more likely to be well predicted.
2020-07-25 06:07:57 +02:00
case CompiledRegex::LineAssertion:
if (not (inst.param.line_start ? is_line_start(pos, config) : is_line_end(pos, config)))
Do not pass thread to failed/consumed, capture it implicitely
2018-12-19 09:16:14 +01:00
return failed();
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
break;
Reduce the amount of Regex VM Instruction code Merge all lookarounds into the same instruction, merge splits, merge literal ignore case with literal... Besides reducing the amount of almost duplicated code, this improves performance by reducing pressure on the (often failing) branch target prediction for instruction dispatching by moving branches into the instruction code themselves where they are more likely to be well predicted.
2020-07-25 06:07:57 +02:00
case CompiledRegex::SubjectAssertion:
if (pos != (inst.param.subject_begin ? config.subject_begin : config.subject_end))
Do not pass thread to failed/consumed, capture it implicitely
2018-12-19 09:16:14 +01:00
return failed();
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
break;
case CompiledRegex::WordBoundary:
Reduce the amount of Regex VM Instruction code Merge all lookarounds into the same instruction, merge splits, merge literal ignore case with literal... Besides reducing the amount of almost duplicated code, this improves performance by reducing pressure on the (often failing) branch target prediction for instruction dispatching by moving branches into the instruction code themselves where they are more likely to be well predicted.
2020-07-25 06:07:57 +02:00
if (is_word_boundary(pos, config) != inst.param.word_boundary_positive)
Do not pass thread to failed/consumed, capture it implicitely
2018-12-19 09:16:14 +01:00
return failed();
Regex: Fix support for ignore case in lookarounds
2017-10-10 05:21:21 +02:00
break;
Reduce the amount of Regex VM Instruction code Merge all lookarounds into the same instruction, merge splits, merge literal ignore case with literal... Besides reducing the amount of almost duplicated code, this improves performance by reducing pressure on the (often failing) branch target prediction for instruction dispatching by moving branches into the instruction code themselves where they are more likely to be well predicted.
2020-07-25 06:07:57 +02:00
case CompiledRegex::LookAround:
if (lookaround(inst.param.lookaround, pos, config) != inst.param.lookaround.positive)
Do not pass thread to failed/consumed, capture it implicitely
2018-12-19 09:16:14 +01:00
return failed();
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
break;
}
}
Do not pass thread to failed/consumed, capture it implicitely
2018-12-19 09:16:14 +01:00
return failed();
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
}
Remove use of utf8::iterator in regex execution This avoids having two copies of the subject string bounds, one in the ExecConfig and one in the utf8 iterator.
2018-11-04 22:17:50 +01:00
bool exec_program(Iterator pos, const ExecConfig& config)
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
{
Regex: Use a custom 'DualThreadStack' structure to hold thread info Instead of using two vectors, we can hold both current and next threads in a single buffer, with stacks growing on each end. Benchmarking shows this to be slightly faster, and should use less memory.
2018-04-29 08:04:26 +02:00
kak_assert(m_threads.current_is_empty() and m_threads.next_is_empty());
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
release_saves(m_captures);
Use indices instead of pointers for saves/instruction in ThreadedRegexVM Performance seems unaffacted, but memory usage should be lowered as the Thread struct is 4 bytes instead of 16.
2018-04-27 00:18:04 +02:00
m_captures = -1;
Never grow the DualThreadStack in push_next As we do at most one push_next per step_thread, and we pop_current before step_thread, we can avoid a branch there at the expense of sometimes growing unecessarily (once).
2018-11-05 21:32:47 +01:00
m_threads.grow_ifn();
Add a peephole optimization pass to the regex compiler
2019-01-06 01:28:39 +01:00
const int16_t first_inst = forward ? 0 : m_program.first_backward_inst;
Refactor regex find next start not to be an instruction anymore The same logic can be hard coded, avoiding one thread and 3 instructions, improving the regex matching speed.
2019-01-05 08:28:56 +01:00
m_threads.push_current({first_inst, -1});
Regex: Support forward and backward matching code in the same CompiledRegex No need to have two separate regexes to handle forward and backward matching, just passing RegexCompileFlags::Backward will add support for backward matching to the regex. For backward only regex, pass RegexCompileFlags::NoForward as well to disable generation of forward matching code.
2017-12-01 12:57:02 +01:00
Remove use of utf8::iterator in regex execution This avoids having two copies of the subject string bounds, one in the ExecConfig and one in the utf8 iterator.
2018-11-04 22:17:50 +01:00
const auto& start_desc = forward ? m_program.forward_start_desc : m_program.backward_start_desc;
Regex: Add a NoSaves RegexExecFlags to disable saving positions
2017-10-03 13:07:44 +02:00
Refactor regex find next start not to be an instruction anymore The same logic can be hard coded, avoiding one thread and 3 instructions, improving the regex matching speed.
2019-01-05 08:28:56 +01:00
constexpr bool search = mode & RegexMode::Search;
Split compile time regex flags from runtime ones
2018-12-27 01:29:33 +01:00
constexpr bool any_match = mode & RegexMode::AnyMatch;
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
uint16_t current_step = -1;
Refactor ThreadedRegexVM::exec_program to avoid branching Moving logic into step_thread instead of returning an enum to select what to run avoids the switch logic and improves run time.
2018-11-05 09:46:53 +01:00
m_found_match = false;
Regex: tag instructions as scheduled as well instead of searching And a few more code cleanup in the ThreadedRegexVM
2017-10-07 13:58:10 +02:00
while (true) // Iterate on all codepoints and once at the end
Regex: refactor ThreadedRegexVM::exec_from code
2017-10-07 13:27:06 +02:00
{
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
if (++current_step == 0)
Regex: remove the need to a processed inst vector Identify each step with a counter, and check if the instruction was already processed this step. This makes the matching faster, by removing the need to maintain a vector of instructions executed this step.
2017-10-14 06:58:42 +02:00
{
// We wrapped, avoid potential collision on inst.last_step by resetting them
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
for (auto& inst : config.instructions)
Regex: remove the need to a processed inst vector Identify each step with a counter, and check if the instruction was already processed this step. This makes the matching faster, by removing the need to maintain a vector of instructions executed this step.
2017-10-14 06:58:42 +02:00
inst.last_step = 0;
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
current_step = 1; // step 0 is never valid
Regex: remove the need to a processed inst vector Identify each step with a counter, and check if the instruction was already processed this step. This makes the matching faster, by removing the need to maintain a vector of instructions executed this step.
2017-10-14 06:58:42 +02:00
}
Regex: Use a custom 'DualThreadStack' structure to hold thread info Instead of using two vectors, we can hold both current and next threads in a single buffer, with stacks growing on each end. Benchmarking shows this to be slightly faster, and should use less memory.
2018-04-29 08:04:26 +02:00
while (not m_threads.current_is_empty())
Refactor ThreadedRegexVM::exec_program to avoid branching Moving logic into step_thread instead of returning an enum to select what to run avoids the switch logic and improves run time.
2018-11-05 09:46:53 +01:00
step_thread(pos, current_step, m_threads.pop_current(), config);
Regex: refactor handling of Saves slightly, do not create them until really needed
2017-10-20 10:55:38 +02:00
Regex: Use a custom 'DualThreadStack' structure to hold thread info Instead of using two vectors, we can hold both current and next threads in a single buffer, with stacks growing on each end. Benchmarking shows this to be slightly faster, and should use less memory.
2018-04-29 08:04:26 +02:00
for (auto& thread : m_threads.next_threads())
Use indices instead of pointers for saves/instruction in ThreadedRegexVM Performance seems unaffacted, but memory usage should be lowered as the Thread struct is 4 bytes instead of 16.
2018-04-27 00:18:04 +02:00
m_program.instructions[thread.inst].scheduled = false;
Regex: Only reset processed and scheduled flags on relevant instructions On big regex, reseting all those flags on all instructions for each character can become the dominant operation. Track that actual instructions index processed (the scheduled are already tracked in the next_threads vector), and only reset these.
2017-10-11 04:24:05 +02:00
Refactor regex find next start not to be an instruction anymore The same logic can be hard coded, avoiding one thread and 3 instructions, improving the regex matching speed.
2019-01-05 08:28:56 +01:00
if (pos == config.end or
(m_threads.next_is_empty() and (not search or m_found_match)) or
(m_found_match and any_match))
Regex: refactor handling of Saves slightly, do not create them until really needed
2017-10-20 10:55:38 +02:00
{
Regex: Use a custom 'DualThreadStack' structure to hold thread info Instead of using two vectors, we can hold both current and next threads in a single buffer, with stacks growing on each end. Benchmarking shows this to be slightly faster, and should use less memory.
2018-04-29 08:04:26 +02:00
for (auto& t : m_threads.next_threads())
Regex: refactor handling of Saves slightly, do not create them until really needed
2017-10-20 10:55:38 +02:00
release_saves(t.saves);
Regex: Use a custom 'DualThreadStack' structure to hold thread info Instead of using two vectors, we can hold both current and next threads in a single buffer, with stacks growing on each end. Benchmarking shows this to be slightly faster, and should use less memory.
2018-04-29 08:04:26 +02:00
m_threads.clear_next();
Refactor ThreadedRegexVM::exec_program to avoid branching Moving logic into step_thread instead of returning an enum to select what to run avoids the switch logic and improves run time.
2018-11-05 09:46:53 +01:00
return m_found_match;
Regex: refactor handling of Saves slightly, do not create them until really needed
2017-10-20 10:55:38 +02:00
}
Regex: Refactor thread handling in ThreadedRegexVM
2017-10-03 12:00:52 +02:00
Remove use of utf8::iterator in regex execution This avoids having two copies of the subject string bounds, one in the ExecConfig and one in the utf8 iterator.
2018-11-04 22:17:50 +01:00
forward ? utf8::to_next(pos, config.subject_end)
: utf8::to_previous(pos, config.subject_begin);
Regex: Go back to instruction based search of next start The previous method, which was a bit faster in the general use case, can hit some cases where we get quadratic behaviour and very slow matching. By using an instruction, we can guarantee our complexity of O(N*M) as we will never have more than N threads (N being the instruction count) and we run the threads once per codepoint in the subject string. That slows down the general case slightly, but ensure we dont have pathological cases. This new version is much faster than the previous instruction based search because it does not use a plain `.*` searcher, but a specific, smarter instruction specialized for finding the next start if we are in the correct conditions.
2017-10-20 09:17:02 +02:00
Fix regex not always selecting the leftmost longest match (Actually the rightmost longest match when searching backwards) Fixes #2710
2019-02-04 07:33:29 +01:00
if (search and not m_found_match)
Refactor regex find next start not to be an instruction anymore The same logic can be hard coded, avoiding one thread and 3 instructions, improving the regex matching speed.
2019-01-05 08:28:56 +01:00
{
if (start_desc and m_threads.next_is_empty())
to_next_start(pos, config, *start_desc);
m_threads.grow_ifn();
m_threads.push_next({first_inst, -1});
}
m_threads.swap_next();
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
}
}
Add static or const where useful
2019-11-02 21:49:26 +01:00
static void to_next_start(Iterator& start, const ExecConfig& config, const StartDesc& start_desc)
Regex: Find potential start position using a map of valid start chars With this optimization we get close to performance parity with boost regex on the common use cases in Kakoune.
2017-10-06 07:40:27 +02:00
{
Remove use of utf8::iterator in regex execution This avoids having two copies of the subject string bounds, one in the ExecConfig and one in the utf8 iterator.
2018-11-04 22:17:50 +01:00
while (start != config.end)
Improve regex vm to next start performance by avoiding iterator copies
2018-10-31 11:13:14 +01:00
{
Do not decode utf-8 when looking for regex next start There is no need to decode as we know any non-ascii characters will be treated as Other in the StartDesc.
2019-12-04 12:00:31 +01:00
static_assert(StartDesc::count <= 128, "start desc should be ascii only");
if constexpr (forward)
Improve regex vm to next start performance by avoiding iterator copies
2018-10-31 11:13:14 +01:00
{
Do not decode utf-8 when looking for regex next start There is no need to decode as we know any non-ascii characters will be treated as Other in the StartDesc.
2019-12-04 12:00:31 +01:00
const unsigned char c = *start;
if (start_desc.map[(c < StartDesc::count) ? c : StartDesc::other])
return;
utf8::to_next(start, config.end);
}
else
{
auto prev = utf8::previous(start, config.end);
const unsigned char c = *prev;
if (start_desc.map[(c < StartDesc::count) ? c : StartDesc::other])
return;
start = prev;
Improve regex vm to next start performance by avoiding iterator copies
2018-10-31 11:13:14 +01:00
}
}
Regex: Find potential start position using a map of valid start chars With this optimization we get close to performance parity with boost regex on the common use cases in Kakoune.
2017-10-06 07:40:27 +02:00
}
Reduce the amount of Regex VM Instruction code Merge all lookarounds into the same instruction, merge splits, merge literal ignore case with literal... Besides reducing the amount of almost duplicated code, this improves performance by reducing pressure on the (often failing) branch target prediction for instruction dispatching by moving branches into the instruction code themselves where they are more likely to be well predicted.
2020-07-25 06:07:57 +02:00
bool lookaround(CompiledRegex::Param::Lookaround param, Iterator pos, const ExecConfig& config) const
Regex: Support any char and character classes in lookarounds Lookarounds still need to be fixed size, but accept character classes as well as plain literals.
2017-10-09 05:12:42 +02:00
{
Cleanup regex lookarounds implementation and reject incompatible regex Fixes #2487
2018-10-10 13:35:38 +02:00
using Lookaround = CompiledRegex::Lookaround;
Reduce the amount of Regex VM Instruction code Merge all lookarounds into the same instruction, merge splits, merge literal ignore case with literal... Besides reducing the amount of almost duplicated code, this improves performance by reducing pressure on the (often failing) branch target prediction for instruction dispatching by moving branches into the instruction code themselves where they are more likely to be well predicted.
2020-07-25 06:07:57 +02:00
if (not param.ahead)
Remove most regex impl special casing for backwards matching
2018-11-03 03:52:40 +01:00
{
if (pos == config.subject_begin)
Reduce the amount of Regex VM Instruction code Merge all lookarounds into the same instruction, merge splits, merge literal ignore case with literal... Besides reducing the amount of almost duplicated code, this improves performance by reducing pressure on the (often failing) branch target prediction for instruction dispatching by moving branches into the instruction code themselves where they are more likely to be well predicted.
2020-07-25 06:07:57 +02:00
return m_program.lookarounds[param.index] == Lookaround::EndOfLookaround;
Remove use of utf8::iterator in regex execution This avoids having two copies of the subject string bounds, one in the ExecConfig and one in the utf8 iterator.
2018-11-04 22:17:50 +01:00
utf8::to_previous(pos, config.subject_begin);
Remove most regex impl special casing for backwards matching
2018-11-03 03:52:40 +01:00
}
Reduce the amount of Regex VM Instruction code Merge all lookarounds into the same instruction, merge splits, merge literal ignore case with literal... Besides reducing the amount of almost duplicated code, this improves performance by reducing pressure on the (often failing) branch target prediction for instruction dispatching by moving branches into the instruction code themselves where they are more likely to be well predicted.
2020-07-25 06:07:57 +02:00
for (auto it = m_program.lookarounds.begin() + param.index; *it != Lookaround::EndOfLookaround; ++it)
Regex: Support any char and character classes in lookarounds Lookarounds still need to be fixed size, but accept character classes as well as plain literals.
2017-10-09 05:12:42 +02:00
{
Reduce the amount of Regex VM Instruction code Merge all lookarounds into the same instruction, merge splits, merge literal ignore case with literal... Besides reducing the amount of almost duplicated code, this improves performance by reducing pressure on the (often failing) branch target prediction for instruction dispatching by moving branches into the instruction code themselves where they are more likely to be well predicted.
2020-07-25 06:07:57 +02:00
if (param.ahead and pos == config.subject_end)
Regex: Support any char and character classes in lookarounds Lookarounds still need to be fixed size, but accept character classes as well as plain literals.
2017-10-09 05:12:42 +02:00
return false;
Remove most regex impl special casing for backwards matching
2018-11-03 03:52:40 +01:00
Remove use of utf8::iterator in regex execution This avoids having two copies of the subject string bounds, one in the ExecConfig and one in the utf8 iterator.
2018-11-04 22:17:50 +01:00
Codepoint cp = utf8::codepoint(pos, config.subject_end);
Reduce the amount of Regex VM Instruction code Merge all lookarounds into the same instruction, merge splits, merge literal ignore case with literal... Besides reducing the amount of almost duplicated code, this improves performance by reducing pressure on the (often failing) branch target prediction for instruction dispatching by moving branches into the instruction code themselves where they are more likely to be well predicted.
2020-07-25 06:07:57 +02:00
if (param.ignore_case)
Regex: Fix support for ignore case in lookarounds
2017-10-10 05:21:21 +02:00
cp = to_lower(cp);
Cleanup regex lookarounds implementation and reject incompatible regex Fixes #2487
2018-10-10 13:35:38 +02:00
const Lookaround op = *it;
if (op == Lookaround::AnyChar)
Regex: Support any char and character classes in lookarounds Lookarounds still need to be fixed size, but accept character classes as well as plain literals.
2017-10-09 05:12:42 +02:00
{} // any character matches
Cleanup regex lookarounds implementation and reject incompatible regex Fixes #2487
2018-10-10 13:35:38 +02:00
else if (op == Lookaround::AnyCharExceptNewLine)
Use a dedicated vm op for dot when match-newline is false
2018-06-24 12:13:35 +02:00
{
if (cp == '\n')
return false;
}
Cleanup regex lookarounds implementation and reject incompatible regex Fixes #2487
2018-10-10 13:35:38 +02:00
else if (op >= Lookaround::CharacterClass and op < Lookaround::CharacterType)
Regex: Replace generic 'Matchers' with specialized functionality Introduce CharacterClass and CharacterType Regex Op, and optimize their evaluation.
2017-11-25 11:14:15 +01:00
{
Cleanup regex lookarounds implementation and reject incompatible regex Fixes #2487
2018-10-10 13:35:38 +02:00
auto index = to_underlying(op) - to_underlying(Lookaround::CharacterClass);
if (not is_character_class(m_program.character_classes[index], cp))
Regex: Replace generic 'Matchers' with specialized functionality Introduce CharacterClass and CharacterType Regex Op, and optimize their evaluation.
2017-11-25 11:14:15 +01:00
return false;
}
Cleanup regex lookarounds implementation and reject incompatible regex Fixes #2487
2018-10-10 13:35:38 +02:00
else if (op >= Lookaround::CharacterType and op < Lookaround::OpEnd)
Regex: Fix lookaround use in moon.kak (?=[A-Z]\w*) is strictly the same as (?=[A-Z]) as \w* will always at least match an empty string.
2017-10-09 05:20:05 +02:00
{
Cleanup regex lookarounds implementation and reject incompatible regex Fixes #2487
2018-10-10 13:35:38 +02:00
auto ctype = static_cast<CharacterType>(to_underlying(op) & 0xFF);
if (not is_ctype(ctype, cp))
Regex: Fix lookaround use in moon.kak (?=[A-Z]\w*) is strictly the same as (?=[A-Z]) as \w* will always at least match an empty string.
2017-10-09 05:20:05 +02:00
return false;
}
Cleanup regex lookarounds implementation and reject incompatible regex Fixes #2487
2018-10-10 13:35:38 +02:00
else if (static_cast<Codepoint>(op) != cp)
Regex: Support any char and character classes in lookarounds Lookarounds still need to be fixed size, but accept character classes as well as plain literals.
2017-10-09 05:12:42 +02:00
return false;
Reduce the amount of Regex VM Instruction code Merge all lookarounds into the same instruction, merge splits, merge literal ignore case with literal... Besides reducing the amount of almost duplicated code, this improves performance by reducing pressure on the (often failing) branch target prediction for instruction dispatching by moving branches into the instruction code themselves where they are more likely to be well predicted.
2020-07-25 06:07:57 +02:00
if (not param.ahead and pos == config.subject_begin)
Remove most regex impl special casing for backwards matching
2018-11-03 03:52:40 +01:00
return *++it == Lookaround::EndOfLookaround;
Reduce the amount of Regex VM Instruction code Merge all lookarounds into the same instruction, merge splits, merge literal ignore case with literal... Besides reducing the amount of almost duplicated code, this improves performance by reducing pressure on the (often failing) branch target prediction for instruction dispatching by moving branches into the instruction code themselves where they are more likely to be well predicted.
2020-07-25 06:07:57 +02:00
param.ahead ? utf8::to_next(pos, config.subject_end)
: utf8::to_previous(pos, config.subject_begin);
Regex: Support any char and character classes in lookarounds Lookarounds still need to be fixed size, but accept character classes as well as plain literals.
2017-10-09 05:12:42 +02:00
}
return true;
}
Remove use of utf8::iterator in regex execution This avoids having two copies of the subject string bounds, one in the ExecConfig and one in the utf8 iterator.
2018-11-04 22:17:50 +01:00
static bool is_line_start(const Iterator& pos, const ExecConfig& config)
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
{
Use custom code instead of reverse_iterator in Regex VM
2018-11-01 22:23:39 +01:00
if (pos == config.subject_begin)
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
return not (config.flags & RegexExecFlags::NotBeginOfLine);
Remove use of utf8::iterator in regex execution This avoids having two copies of the subject string bounds, one in the ExecConfig and one in the utf8 iterator.
2018-11-04 22:17:50 +01:00
return utf8::codepoint(utf8::previous(pos, config.subject_begin), config.subject_end) == '\n';
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
}
Remove use of utf8::iterator in regex execution This avoids having two copies of the subject string bounds, one in the ExecConfig and one in the utf8 iterator.
2018-11-04 22:17:50 +01:00
static bool is_line_end(const Iterator& pos, const ExecConfig& config)
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
{
Use custom code instead of reverse_iterator in Regex VM
2018-11-01 22:23:39 +01:00
if (pos == config.subject_end)
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
return not (config.flags & RegexExecFlags::NotEndOfLine);
Remove use of utf8::iterator in regex execution This avoids having two copies of the subject string bounds, one in the ExecConfig and one in the utf8 iterator.
2018-11-04 22:17:50 +01:00
return utf8::codepoint(pos, config.subject_end) == '\n';
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
}
Remove use of utf8::iterator in regex execution This avoids having two copies of the subject string bounds, one in the ExecConfig and one in the utf8 iterator.
2018-11-04 22:17:50 +01:00
static bool is_word_boundary(const Iterator& pos, const ExecConfig& config)
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
{
Use custom code instead of reverse_iterator in Regex VM
2018-11-01 22:23:39 +01:00
if (pos == config.subject_begin)
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
return not (config.flags & RegexExecFlags::NotBeginOfWord);
Use custom code instead of reverse_iterator in Regex VM
2018-11-01 22:23:39 +01:00
if (pos == config.subject_end)
Regex: Refactor ThreadedRegexVM state handling Remove ExecState to store threads inside the ThreadedRegexVM so that memory buffers can be reused between executions. Extract an ExecConfig struct with all the data thats execution specific to avoid storing it needlessly inside the ThreadedRegexVM.
2018-04-21 04:44:54 +02:00
return not (config.flags & RegexExecFlags::NotEndOfWord);
Remove use of utf8::iterator in regex execution This avoids having two copies of the subject string bounds, one in the ExecConfig and one in the utf8 iterator.
2018-11-04 22:17:50 +01:00
return is_word(utf8::codepoint(utf8::previous(pos, config.subject_begin), config.subject_end)) !=
is_word(utf8::codepoint(pos, config.subject_end));
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
}
Remove use of utf8::iterator in regex execution This avoids having two copies of the subject string bounds, one in the ExecConfig and one in the utf8 iterator.
2018-11-04 22:17:50 +01:00
static Codepoint codepoint(const Iterator& it, const ExecConfig& config)
{
return utf8::codepoint(forward ? it : utf8::previous(it, config.subject_begin),
config.subject_end);
}
Regex: Add support for backward matching Regex can be compiled for backward matching instead of forward matching and the ThreadedRegexVM is able to iterate in reverse on the subject string to find the last match instead of the first.
2017-10-07 06:46:27 +02:00
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
const CompiledRegex& m_program;
Regex: Use a custom 'DualThreadStack' structure to hold thread info Instead of using two vectors, we can hold both current and next threads in a single buffer, with stacks growing on each end. Benchmarking shows this to be slightly faster, and should use less memory.
2018-04-29 08:04:26 +02:00
struct DualThreadStack
{
Various micro performance tweaks in ThreadedRegexVM
2018-11-05 11:54:29 +01:00
DualThreadStack() = default;
DualThreadStack(const DualThreadStack&) = delete;
~DualThreadStack() { delete[] m_data; }
Regex: Use a custom 'DualThreadStack' structure to hold thread info Instead of using two vectors, we can hold both current and next threads in a single buffer, with stacks growing on each end. Benchmarking shows this to be slightly faster, and should use less memory.
2018-04-29 08:04:26 +02:00
bool current_is_empty() const { return m_current == 0; }
bool next_is_empty() const { return m_next == m_capacity; }
Never grow the DualThreadStack in push_next As we do at most one push_next per step_thread, and we pop_current before step_thread, we can avoid a branch there at the expense of sometimes growing unecessarily (once).
2018-11-05 21:32:47 +01:00
void push_current(Thread thread) { kak_assert(m_current < m_next); m_data[m_current++] = thread; grow_ifn(); }
Regex: Use a custom 'DualThreadStack' structure to hold thread info Instead of using two vectors, we can hold both current and next threads in a single buffer, with stacks growing on each end. Benchmarking shows this to be slightly faster, and should use less memory.
2018-04-29 08:04:26 +02:00
Thread pop_current() { kak_assert(m_current > 0); return m_data[--m_current]; }
Never grow the DualThreadStack in push_next As we do at most one push_next per step_thread, and we pop_current before step_thread, we can avoid a branch there at the expense of sometimes growing unecessarily (once).
2018-11-05 21:32:47 +01:00
void push_next(Thread thread) { kak_assert(m_current < m_next); m_data[--m_next] = thread; }
Regex: Use a custom 'DualThreadStack' structure to hold thread info Instead of using two vectors, we can hold both current and next threads in a single buffer, with stacks growing on each end. Benchmarking shows this to be slightly faster, and should use less memory.
2018-04-29 08:04:26 +02:00
void clear_next() { m_next = m_capacity; }
Various micro performance tweaks in ThreadedRegexVM
2018-11-05 11:54:29 +01:00
ConstArrayView<Thread> next_threads() const { return { m_data + m_next, m_data + m_capacity }; }
Regex: Use a custom 'DualThreadStack' structure to hold thread info Instead of using two vectors, we can hold both current and next threads in a single buffer, with stacks growing on each end. Benchmarking shows this to be slightly faster, and should use less memory.
2018-04-29 08:04:26 +02:00
void swap_next()
{
Various micro performance tweaks in ThreadedRegexVM
2018-11-05 11:54:29 +01:00
kak_assert(m_next < m_capacity);
Small code tweak in DualThreadStack::swap_next
2018-11-14 07:50:17 +01:00
const int32_t count = m_capacity - m_next;
std::copy_n(m_data + m_next, count, m_data);
m_current = count;
m_next = m_capacity;
Regex: Use a custom 'DualThreadStack' structure to hold thread info Instead of using two vectors, we can hold both current and next threads in a single buffer, with stacks growing on each end. Benchmarking shows this to be slightly faster, and should use less memory.
2018-04-29 08:04:26 +02:00
}
void grow_ifn()
{
if (m_current != m_next)
return;
Raise ThreadedVM initial thread capacity to 16 Threads are 4 bytes, an initial capacity of 4 led to allocating 16 bytes, raising that to 64 bytes seems quite reasonable.
2021-03-03 10:51:24 +01:00
constexpr int32_t initial_capacity = 64 / sizeof(Thread);
static_assert(initial_capacity >= 4);
const auto new_capacity = m_capacity ? m_capacity * 2 : initial_capacity;
Fix memory leak in DualThreadStack Fixes #2556
2018-11-07 02:28:41 +01:00
const auto next_count = m_capacity - m_next;
const auto new_next = new_capacity - next_count;
Regex: Use a custom 'DualThreadStack' structure to hold thread info Instead of using two vectors, we can hold both current and next threads in a single buffer, with stacks growing on each end. Benchmarking shows this to be slightly faster, and should use less memory.
2018-04-29 08:04:26 +02:00
Thread* new_data = new Thread[new_capacity];
Fix memory leak in DualThreadStack Fixes #2556
2018-11-07 02:28:41 +01:00
std::copy_n(m_data, m_current, new_data);
std::copy_n(m_data + m_next, next_count, new_data + new_next);
delete[] m_data;
Various micro performance tweaks in ThreadedRegexVM
2018-11-05 11:54:29 +01:00
m_data = new_data;
Regex: Use a custom 'DualThreadStack' structure to hold thread info Instead of using two vectors, we can hold both current and next threads in a single buffer, with stacks growing on each end. Benchmarking shows this to be slightly faster, and should use less memory.
2018-04-29 08:04:26 +02:00
m_capacity = new_capacity;
m_next = new_next;
}
Never grow the DualThreadStack in push_next As we do at most one push_next per step_thread, and we pop_current before step_thread, we can avoid a branch there at the expense of sometimes growing unecessarily (once).
2018-11-05 21:32:47 +01:00
private:
Various micro performance tweaks in ThreadedRegexVM
2018-11-05 11:54:29 +01:00
Thread* m_data = nullptr;
Fix integer overflow leading to bad memory access in regex execution Fixes #2481 Fixes #2480
2018-10-08 03:43:03 +02:00
int32_t m_capacity = 0; // Maximum capacity should be 2*instruction count, so 65536
int32_t m_current = 0;
int32_t m_next = 0;
Regex: Use a custom 'DualThreadStack' structure to hold thread info Instead of using two vectors, we can hold both current and next threads in a single buffer, with stacks growing on each end. Benchmarking shows this to be slightly faster, and should use less memory.
2018-04-29 08:04:26 +02:00
};
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
Split compile time regex flags from runtime ones
2018-12-27 01:29:33 +01:00
static_assert(has_direction(mode));
static constexpr bool forward = mode & RegexMode::Forward;
Remove use of utf8::iterator in regex execution This avoids having two copies of the subject string bounds, one in the ExecConfig and one in the utf8 iterator.
2018-11-04 22:17:50 +01:00
Regex: Use a custom 'DualThreadStack' structure to hold thread info Instead of using two vectors, we can hold both current and next threads in a single buffer, with stacks growing on each end. Benchmarking shows this to be slightly faster, and should use less memory.
2018-04-29 08:04:26 +02:00
DualThreadStack m_threads;
Regex: Introduce a Regex memory domain to track usage separately
2017-10-15 03:23:57 +02:00
Vector<Saves*, MemoryDomain::Regex> m_saves;
Use indices instead of pointers for saves/instruction in ThreadedRegexVM Performance seems unaffacted, but memory usage should be lowered as the Thread struct is 4 bytes instead of 16.
2018-04-27 00:18:04 +02:00
int16_t m_first_free = -1;
int16_t m_captures = -1;
Refactor ThreadedRegexVM::exec_program to avoid branching Moving logic into step_thread instead of returning an enum to select what to run avoids the switch logic and improves run time.
2018-11-05 09:46:53 +01:00
bool m_found_match = false;
Regex: validate that our custom impl gets the same results as boost regex In addition to running boost regex, run our custom regex and compare the results to ensure the two regex engine agree.
2017-10-02 08:59:04 +02:00
};
Regex: whenever Kakoune compiles a regex, pass it to the custom impl as well That way we can see which features are missing.
2017-09-26 09:44:30 +02:00
}
Regex: Add initial, exploratory work on a custom regex engine
2017-09-17 10:50:53 +02:00
#endif // regex_impl_hh_INCLUDED
Reference in New Issue Copy Permalink