{“payload”:{“allShortcutsEnabled”:false,”fileTree”:{“”:{“items”:[{“name”:”.github”,”path”:”.github”,”contentType”:”directory”},{“name”:”alt”,”path”:”alt”,”contentType”:”directory”},{“name”:”asm”,”path”:”asm”,”contentType”:”directory”},{“name”:”assets”,”path”:”assets”,”contentType”:”directory”},{“name”:”cmd”,”path”:”cmd”,”contentType”:”directory”},{“name”:”gen”,”path”:”gen”,”contentType”:”directory”},{“name”:”jp”,”path”:”jp”,”contentType”:”directory”},{“name”:”oj”,”path”:”oj”,”contentType”:”directory”},{“name”:”pretty”,”path”:”pretty”,”contentType”:”directory”},{“name”:”sen”,”path”:”sen”,”contentType”:”directory”},{“name”:”tt”,”path”:”tt”,”contentType”:”directory”},{“name”:”.coveralls.yml”,”path”:”.coveralls.yml”,”contentType”:”file”},{“name”:”.gitignore”,”path”:”.gitignore”,”contentType”:”file”},{“name”:”.golangci.yml”,”path”:”.golangci.yml”,”contentType”:”file”},{“name”:”CHANGELOG.md”,”path”:”CHANGELOG.md”,”contentType”:”file”},{“name”:”LICENSE”,”path”:”LICENSE”,”contentType”:”file”},{“name”:”Makefile”,”path”:”Makefile”,”contentType”:”file”},{“name”:”README.md”,”path”:”README.md”,”contentType”:”file”},{“name”:”benchmarks.md”,”path”:”benchmarks.md”,”contentType”:”file”},{“name”:”converter.go”,”path”:”converter.go”,”contentType”:”file”},{“name”:”converter_test.go”,”path”:”converter_test.go”,”contentType”:”file”},{“name”:”design.md”,”path”:”design.md”,”contentType”:”file”},{“name”:”doc.go”,”path”:”doc.go”,”contentType”:”file”},{“name”:”error.go”,”path”:”error.go”,”contentType”:”file”},{“name”:”error_test.go”,”path”:”error_test.go”,”contentType”:”file”},{“name”:”go.mod”,”path”:”go.mod”,”contentType”:”file”},{“name”:”go.sum”,”path”:”go.sum”,”contentType”:”file”},{“name”:”go.yml”,”path”:”go.yml”,”contentType”:”file”},{“name”:”notes”,”path”:”notes”,”contentType”:”file”},{“name”:”numconvmethod.go”,”path”:”numconvmethod.go”,”contentType”:”file”},{“name”:”options.go”,”path”:”options.go”,”contentType”:”file”},{“name”:”options_test.go”,”path”:”options_test.go”,”contentType”:”file”},{“name”:”sen.md”,”path”:”sen.md”,”contentType”:”file”},{“name”:”string.go”,”path”:”string.go”,”contentType”:”file”},{“name”:”string_test.go”,”path”:”string_test.go”,”contentType”:”file”}],”totalCount”:35}},”fileTreeProcessingTime”:5.670610000000001,”foldersToFetch”:[],”reducedMotionEnabled”:null,”repo”:{“id”:255132844,”defaultBranch”:”develop”,”name”:”ojg”,”ownerLogin”:”ohler55″,”currentUserCanPush”:false,”isFork”:false,”isEmpty”:false,”createdAt”:”2020-04-12T17:17:31.000Z”,”ownerAvatar”:”https://avatars.githubusercontent.com/u/118856?v=4″,”public”:true,”private”:false,”isOrgOwned”:false},”symbolsExpanded”:false,”treeExpanded”:true,”refInfo”:{“name”:”develop”,”listCacheKey”:”v0:1696374237.0″,”canEdit”:false,”refType”:”branch”,”currentOid”:”f010b727b0f875b848d8d84e2eea889344371e54″},”path”:”design.md”,”currentUser”:null,”blob”:{“rawLines”:null,”stylingDirectives”:null,”csv”:null,”csvError”:null,”dependabotInfo”:{“showConfigurationBanner”:false,”configFilePath”:null,”networkDependabotPath”:”/ohler55/ojg/network/updates”,”dismissConfigurationNoticePath”:”/settings/dismiss-notice/dependabot_configuration_notice”,”configurationNoticeDismissed”:null,”repoAlertsPath”:”/ohler55/ojg/security/dependabot”,”repoSecurityAndAnalysisPath”:”/ohler55/ojg/settings/security_analysis”,”repoOwnerIsOrg”:false,”currentUserCanAdminRepo”:false},”displayName”:”design.md”,”displayUrl”:”https://github.com/ohler55/ojg/blob/develop/design.md?raw=true”,”headerInfo”:{“blobSize”:”23.2 KB”,”deleteInfo”:{“deleteTooltip”:”You must be signed in to make or propose changes”},”editInfo”:{“editTooltip”:”You must be signed in to make or propose changes”},”ghDesktopPath”:”https://desktop.github.com”,”gitLfsPath”:null,”onBranch”:true,”shortPath”:”21c3bfe”,”siteNavLoginPath”:”/login?return_to=https%3A%2F%2Fgithub.com%2Fohler55%2Fojg%2Fblob%2Fdevelop%2Fdesign.md”,”isCSV”:false,”isRichtext”:true,”toc”:[{“level”:1,”text”:”A Journey building a fast JSON parser and full JSONPath, Oj for Go”,”anchor”:”a-journey-building-a-fast-json-parser-and-full-jsonpath-oj-for-go”,”htmlText”:”A Journey building a fast JSON parser and full JSONPath, Oj for Go”},{“level”:2,”text”:”Planning”,”anchor”:”planning”,”htmlText”:”Planning”},{“level”:3,”text”:”Generic Data”,”anchor”:”generic-data”,”htmlText”:”Generic Data”},{“level”:3,”text”:”JSON Parser and Validator”,”anchor”:”json-parser-and-validator”,”htmlText”:”JSON Parser and Validator”},{“level”:3,”text”:”JSONPath”,”anchor”:”jsonpath”,”htmlText”:”JSONPath”},{“level”:2,”text”:”The Journey”,”anchor”:”the-journey”,”htmlText”:”The Journey”},{“level”:3,”text”:”Generic Data (gen package)”,”anchor”:”generic-data-gen-package”,”htmlText”:”Generic Data (gen package)”},{“level”:3,”text”:”Simple Parser (oj package)”,”anchor”:”simple-parser-oj-package”,”htmlText”:”Simple Parser (oj package)”},{“level”:4,”text”:”Validator”,”anchor”:”validator”,”htmlText”:”Validator”},{“level”:4,”text”:”Parser”,”anchor”:”parser”,”htmlText”:”Parser”},{“level”:3,”text”:”JSONPath (jp package)”,”anchor”:”jsonpath-jp-package”,”htmlText”:”JSONPath (jp package)”},{“level”:3,”text”:”Converting or Altering Data (alt package)”,”anchor”:”converting-or-altering-data-alt-package”,”htmlText”:”Converting or Altering Data (alt package)”},{“level”:2,”text”:”Lessons Learned”,”anchor”:”lessons-learned”,”htmlText”:”Lessons Learned”},{“level”:3,”text”:”Functions Add Overhead”,”anchor”:”functions-add-overhead”,”htmlText”:”Functions Add Overhead”},{“level”:3,”text”:”Slices are Nice”,”anchor”:”slices-are-nice”,”htmlText”:”Slices are Nice”},{“level”:3,”text”:”Memory Allocation”,”anchor”:”memory-allocation”,”htmlText”:”Memory Allocation”},{“level”:3,”text”:”Range Has Been Optimized”,”anchor”:”range-has-been-optimized”,”htmlText”:”Range Has Been Optimized”},{“level”:3,”text”:”APIs Matter”,”anchor”:”apis-matter”,”htmlText”:”APIs Matter”},{“level”:2,”text”:”Whats Next?”,”anchor”:”whats-next”,”htmlText”:”Whats Next?”}],”lineInfo”:{“truncatedLoc”:”500″,”truncatedSloc”:”408″},”mode”:”file”},”image”:false,”isCodeownersFile”:null,”isPlain”:false,”isValidLegacyIssueTemplate”:false,”issueTemplateHelpUrl”:”https://docs.github.com/articles/about-issue-and-pull-request-templates”,”issueTemplate”:null,”discussionTemplate”:null,”language”:”Markdown”,”languageID”:222,”large”:false,”loggedIn”:false,”newDiscussionPath”:”/ohler55/ojg/discussions/new”,”newIssuePath”:”/ohler55/ojg/issues/new”,”planSupportInfo”:{“repoIsFork”:null,”repoOwnedByCurrentUser”:null,”requestFullPath”:”/ohler55/ojg/blob/develop/design.md”,”showFreeOrgGatedFeatureMessage”:null,”showPlanSupportBanner”:null,”upgradeDataAttributes”:null,”upgradePath”:null},”publishBannersInfo”:{“dismissActionNoticePath”:”/settings/dismiss-notice/publish_action_from_dockerfile”,”dismissStackNoticePath”:”/settings/dismiss-notice/publish_stack_from_file”,”releasePath”:”/ohler55/ojg/releases/new?marketplace=true”,”showPublishActionBanner”:false,”showPublishStackBanner”:false},”rawBlobUrl”:”https://github.com/ohler55/ojg/raw/develop/design.md”,”renderImageOrRaw”:false,”richText”:”
I had a dream. I’d write a fast JSON parser, generic data, and anJSONPath implementation and it would be beautiful, well organized, andnsomething to be admired. Well, reality kicked in and laughed at thosendreams. A Go JSON parser and tools could be high performance but tonget that performance compromises in beauty would have to be made. Thisnis a tale of journey that ended with a Parser that leaves the Go JSONnparser in the dust and resulted in some useful tools including ancomplete and efficient JSONPath implementation.
n
In all fairness I did embark on with some previous experience. Havingnwritten two JSON parser before. Both the RubynOj and the C parsernOjC. Why not annOjG for go.
n
Planning
n
Like any journey it starts with the planning. Yeah, I know, it’s callednrequirement gathering but casting it as planning a journey is more funnand this was all about enjoying the discoveries on the journey. Thenjourney takes place in the land of OjG which stands for Oj fornGo. Oj or Optimized JSON being anpopular gem I wrote for Ruby.
n
First, JSON parsing and any frequently used operations such asnJSONPath evaluation had to be fast over everything else. With thenluxury of not having to follow the existing Go json package API thenAPI could be designed for the best performance.
n
The journey would visit several areas each with its own landscape andndifferent problems to solve.
n
Generic Data
n
The first visit was to generic data. Not to be confused with thenproposed Go generics. Thats a completely different animal and hasnnothing to do with whats being referred to as generic data here. Innbuilding tools or packages for reuse the data acted on by those toolsnneeds to be navigable.
n
Reflection can be used but that gets a bit tricky when dealing withnprivate fields or field that can’t be converted to something that cannsay be written as a JSON element. Other options are often better.
n
Another approach is to use simple Go types such as bool, int64,n[]any, and other types that map directly on to JSON or somenother subset of all possible Go types. If too open, such as withn[]any it is still possible for the user to put unsupportedntypes into the data. Not to pick out any package specifically but itnis frustrating to see an argument type of any in an API andnthen no documentation describing that the supported types are.
n
There is another approach though: Define a set of types that can be inna collection and use those types. With this approach, the generic datanimplementation has to support the basic JSON types of null,nboolean, int64, float64, string, array, and object. Innaddition time should be supported. From experience in both JSON use innRuby and Go time has always been needed. Time is just too much a partnof any set of data to leave it out.
n
The generic data had to be type safe. It would not do to have annelement that could not be encoded as JSON in the data.
n
A frequent operation for generic data is to store that data into anJSON database or similar. That meant converting to simple Go types ofnnil, bool, int64, float64, string, []any, andnmap[string]any had to be fast.
n
Also planned for this part of the journey was methods on the types tonsupport getting, setting, and deleting elements using JSONPath. Thenhope was to have an object based approach to the generic nodes sonsomething like the following could be used but keeping generic data,nJSONPath, and parsing in separate packages.
n
var n gen.Noden n = gen.Int(123)n i, ok := n.AsInt()
n
Unfortunately that part of the journey had to be cancelled as the Gontravel guide refuses to let packages talk back and forth. Imports arenone way only. After trying to put all the code in one package itneventually got unwieldy. Function names started being prefixed withnwhat should really have been package names so the object and methodnapproach was dropped. A change in API but the journey would continue.
n
JSON Parser and Validator
n
The next stop was the parser and validator. After some considerationnit seemed like starting with the validator would be best way to becomenfamiliar with the territory. The JSON parser and validator need not benthe same and each should be as performant as possible. The parsersnneeded to support parsing into simple Go types as well as the genericndata types.
n
When parsing files that include millions or more JSON elements innfiles that might be over 100GB a streaming parser is necessary. Itnwould be nice to share some code with both the streaming and stringnparsers of course. It’s easier to pack light when the areas arensimilar.
n
The parser must also allow parsing into native Go types. Furthermoreninterfaces must be supported even though Go unmarshalling does notnsupport interface fields. Many data types make use of interfacesnthat limitation was not acceptable for the OjG parser. A differentnapproach to support interfaces was possible.
n
JSON documents of any non-trivial size, especially if hand-edited, arenlikely to have errors at some point. Parse errors must identify wherenin the document the error occurred.
n
JSONPath
n
Saving the most interesting part of the trip for last, the JSONPathnimplementation promised to have all sorts of interesting problems tonsolve with descents, wildcards, and especially filters.
n
A JSONPath is used to extract elements from data. That part of thenimplementation had to be fast. Parsing really didn’t have to be fastnbut it would be nice to have a way of building a JSONPath in anperformant manner even if it was not as convenient as parsing anstring.
n
The JSONPath implementation had to implement all the featuresndescribed by the Goessnernarticle. There are otherndescriptions of JSONPath but the Goessner description is the mostnreferenced. Since the implementation is in Go the scripting featurendescribed could be left out as long as similar functionality could benprovided for array indexes relative to the length of thenarray. Borrowing from Ruby, using negative indexes would provide thatnfunctionality.
n
The Journey
n
The journey unfolded as planned to a degree. There were some falsenstarts and revisits but eventually each destination was reached andnthe journey completed.
n
Generic Data (gen package)
n
What better way to make generic type fast than to just define genericntypes from simple Go types and then add methods on those types? Angen.Int is just an int64 and a gen.Array is just an[]gen.Node. With that approach there are no extra allocations.
n
type Node anyntype Int int64ntype Array []Node
n
Since generic arrays and objects restrict the type of the values inneach collection to gen.Node types the collections are assured toncontain only elements that can be encoded as JSON.
n
Methods on the Node could not be imp
