{"id":3448,"date":"2018-05-08T11:30:52","date_gmt":"2018-05-08T10:30:52","guid":{"rendered":"http:\/\/roboblog.fatal-fury.de\/?p=3448"},"modified":"2018-05-08T13:22:34","modified_gmt":"2018-05-08T12:22:34","slug":"c-guns-tokenizer","status":"publish","type":"post","link":"http:\/\/roboblog.fatal-fury.de\/?p=3448","title":{"rendered":"C++ Guns: tokenizer"},"content":{"rendered":"

Hier mal ein Versuch ein h\u00fcbscheren tokenizer zu bauen:<\/p>\n

\r\n\/\/\/ \\todo TH use string_view\r\nstd::list<std::string> tokenize(const std::string& str) {\r\n    std::list<std::string> tokens;\r\n    std::string token;\r\n\r\n    auto add = [&tokens](std::string& token) {\r\n        if(not token.empty()) {\r\n            tokens.emplace_back(std::move(token));\r\n        }\r\n    };\r\n\r\n    for(const char c : str) {\r\n        if(c == ' ') {\r\n            add(token);\r\n        } else if(c == '(' or c == ')') {\r\n            add (token);\r\n            tokens.emplace_back(std::string{c});\r\n        } else {\r\n            token.push_back(c);\r\n        }\r\n    }\r\n\r\n    add(token);\r\n\r\n    return tokens;\r\n}\r\n<\/pre>\n
Ist auf alle F\u00e4lle Millionen mal besser als Dieses hier:
\nhttps:\/\/gist.github.com\/ofan\/721464<\/p>\n
\r\nstd::list<std::string> tokenize(const std::string & str)\r\n{\r\n    std::list<std::string> tokens;\r\n    const char * s = str.c_str();\r\n    while (*s) {\r\n        while (*s == ' ')\r\n            ++s;\r\n        if (*s == '(' || *s == ')')\r\n            tokens.push_back(*s++ == '(' ? "(" : ")");\r\n        else {\r\n            const char * t = s;\r\n            while (*t && *t != ' ' && *t != '(' && *t != ')')\r\n                ++t;\r\n            tokens.push_back(std::string(s, t));\r\n            s = t;\r\n        }\r\n    }\r\n    return tokens;\r\n}\r\n<\/pre>\n
hf<\/p>\n","protected":false},"excerpt":{"rendered":"
Hier mal ein Versuch ein h\u00fcbscheren tokenizer zu bauen: \/\/\/ \\todo TH use string_view std::list<std::string> tokenize(const std::string& str) { std::list<std::string> tokens; std::string token; auto add = [&tokens](std::string& token) { if(not token.empty()) { tokens.emplace_back(std::move(token)); } }; for(const char c : str) { if(c == ' ') { add(token); } else if(c == '(' or c == […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4],"tags":[17],"class_list":["post-3448","post","type-post","status-publish","format-standard","hentry","category-allgemein","tag-cpp"],"_links":{"self":[{"href":"http:\/\/roboblog.fatal-fury.de\/index.php?rest_route=\/wp\/v2\/posts\/3448","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/roboblog.fatal-fury.de\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/roboblog.fatal-fury.de\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/roboblog.fatal-fury.de\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"http:\/\/roboblog.fatal-fury.de\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=3448"}],"version-history":[{"count":3,"href":"http:\/\/roboblog.fatal-fury.de\/index.php?rest_route=\/wp\/v2\/posts\/3448\/revisions"}],"predecessor-version":[{"id":3451,"href":"http:\/\/roboblog.fatal-fury.de\/index.php?rest_route=\/wp\/v2\/posts\/3448\/revisions\/3451"}],"wp:attachment":[{"href":"http:\/\/roboblog.fatal-fury.de\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=3448"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/roboblog.fatal-fury.de\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=3448"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/roboblog.fatal-fury.de\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=3448"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}