Compare commits

..

15 Commits

Author SHA1 Message Date
Lunny Xiao
cf217befb0 Fix database keyword quote problem on migration v161 (#17524)
* Fix database keyword quote problem on migration v161

* support rerun migration v161
2021-11-03 08:28:21 +08:00
6543
f8503b5fbf Fix bug of migrated repository not index (#16991) (#16995)
Fix #16986, #16152

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2021-09-08 22:32:54 +01:00
zeripath
0e53baf6d1 Add Changelog for 1.14.7 (#16924)
## [1.14.7](https://github.com/go-gitea/gitea/releases/tag/v1.14.7) - 2021-09-02

* BUGFIXES
  * Add missing gitRepo close at GetDiffRangeWithWhitespaceBehavior (Partial #16894) (#16896)
  * Fix wiki raw commit diff/patch view (#16891) (#16893)
  * Ensure wiki repos are all closed (#16886) (#16889)
  * Upgrade xorm to v1.2.2 (#16663) & Add test to ensure that dumping of login sources remains correct (#16847) (#16849)
  * Recreate Tables should Recreate indexes on MySQL (#16718) (#16740)

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-09-02 17:39:33 +01:00
6543
1f7187c973 Add missing gitRepo close (#16896) 2021-08-31 14:12:37 +08:00
6543
2fc5b08c42 Fix wiki raw commit diff/patch view (#16891) (#16893) 2021-08-31 05:01:15 +02:00
6543
95d2744020 Ensure wiki repos are all closed (#16886) (#16889)
There are multiple places where wiki git repositories are not properly closed.

This PR ensures they are closed.

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Andrew Thornton <art27@cantab.net>
2021-08-30 23:22:36 +02:00
zeripath
73e5c36f25 Upgrade xorm to v1.2.2 (#16663) & Add test to ensure that dumping of login sources remains correct (#16847) (#16849)
* Upgrade xorm to v1.2.2 (#16663)

Backport #16663

Fix #16683

* Upgrade xorm to v1.2.2

* Change the Engine interface to match xorm v1.2.2

* Add test to ensure that dumping of login sources remains correct (#16847)

#16831 has occurred because of a missed regression. This PR adds a simple test to
try to prevent this occuring again.

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2021-08-28 13:15:21 +02:00
zeripath
bc1fefce87 Recreate Tables should Recreate indexes on MySQL (#16718) (#16740)
Backport #16718

The MySQL indexes are not being renamed at the same time as RENAME table despite the
CASCADE. Therefore it is probably better to just recreate the indexes instead.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-08-20 22:17:07 -04:00
zeripath
bb054fdfa1 Changelog for 1.14.6 (#16619)
## [1.14.6](https://github.com/go-gitea/gitea/releases/tag/v1.14.6) - 2021-08-04

* SECURITY
  * Bump github.com/markbates/goth from v1.67.1 to v1.68.0 (#16538) (#16540)
  * Switch to maintained JWT lib (#16532) (#16535)
  * Upgrade to latest version of golang-jwt (as forked for 1.14) (#16590) (#16607)
* BUGFIXES
  * Add basic edit ldap auth test & actually fix #16252 (#16465) (#16495)
  * Make cancel from CatFileBatch and CatFileBatchCheck wait for the command to end (#16479) (#16481)

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-08-05 17:39:25 +01:00
zeripath
7760a7f385 Upgrade to latest version of golang-jwt (as forked for 1.14) (#16607)
* Forcibly update the vendored versions using a replace

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-08-04 00:21:00 +03:00
6543
3107c9dfc3 upgraded github.com/markbates/goth v1.67.1 => v1.68.0 (#16540) 2021-07-24 18:16:07 +01:00
6543
a66ff8a210 switch to maintained lib (#16535)
Co-authored-by: Matti R <matti@mdranta.net>
2021-07-24 11:13:56 -04:00
zeripath
6a3c7856c8 Add basic edit ldap auth test & actually fix #16252 (#16465) (#16495)
Backport #16465

One of the reasons why #16447 was needed and why #16268 was needed in
the first place was because it appears that editing ldap configuration
doesn't get tested.

This PR therefore adds a basic test that will run the edit pipeline.

In doing so it's now clear that #16447 and #16268 aren't actually
solving #16252. It turns out that what actually happens is that is that
the bytes are actually double encoded.

This PR now changes the json unmarshal wrapper to handle this double
encode.

Fix #16252

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: 6543 <6543@obermui.de>
2021-07-22 16:33:50 +03:00
zeripath
3299f044d3 Make cancel from CatFileBatch and CatFileBatchCheck wait for the command to end (#16479) (#16481)
* Make cancel from CatFileBatch and CatFileBatchCheck wait for the command to end

Fix #16427 (again!)

* handle sharing violation error code

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-07-21 18:34:53 +02:00
zeripath
e6c222511d Retry rename on lock induced failures (re-fix) (#16461) (#16463)
Backport #16461

Unfortunately #16435 asserts the wrong error and should use
os.LinkError not os.PathError.

Fix #16439

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-07-16 17:48:39 +01:00
413 changed files with 55849 additions and 25468 deletions

View File

@@ -4,6 +4,25 @@ This changelog goes through all the changes that have been made in each release
without substantial changes to our git log; to see the highlights of what has
been added to each release, please refer to the [blog](https://blog.gitea.io).
## [1.14.7](https://github.com/go-gitea/gitea/releases/tag/v1.14.7) - 2021-09-02
* BUGFIXES
* Add missing gitRepo close at GetDiffRangeWithWhitespaceBehavior (Partial #16894) (#16896)
* Fix wiki raw commit diff/patch view (#16891) (#16893)
* Ensure wiki repos are all closed (#16886) (#16889)
* Upgrade xorm to v1.2.2 (#16663) & Add test to ensure that dumping of login sources remains correct (#16847) (#16849)
* Recreate Tables should Recreate indexes on MySQL (#16718) (#16740)
## [1.14.6](https://github.com/go-gitea/gitea/releases/tag/v1.14.6) - 2021-08-04
* SECURITY
* Bump github.com/markbates/goth from v1.67.1 to v1.68.0 (#16538) (#16540)
* Switch to maintained JWT lib (#16532) (#16535)
* Upgrade to latest version of golang-jwt (as forked for 1.14) (#16590) (#16607)
* BUGFIXES
* Add basic edit ldap auth test & actually fix #16252 (#16465) (#16495)
* Make cancel from CatFileBatch and CatFileBatchCheck wait for the command to end (#16479) (#16481)
## [1.14.5](https://github.com/go-gitea/gitea/releases/tag/v1.14.5) - 2021-07-16
* SECURITY

View File

@@ -23,7 +23,7 @@ import (
"code.gitea.io/gitea/modules/private"
"code.gitea.io/gitea/modules/setting"
"github.com/dgrijalva/jwt-go"
"github.com/golang-jwt/jwt"
jsoniter "github.com/json-iterator/go"
"github.com/kballard/go-shellquote"
"github.com/urfave/cli"

27
go.mod
View File

@@ -27,8 +27,7 @@ require (
github.com/couchbase/gomemcached v0.1.2 // indirect
github.com/couchbase/goutils v0.0.0-20210118111533-e33d3ffb5401 // indirect
github.com/cpuguy83/go-md2man/v2 v2.0.0 // indirect
github.com/denisenkom/go-mssqldb v0.9.0
github.com/dgrijalva/jwt-go v3.2.0+incompatible
github.com/denisenkom/go-mssqldb v0.10.0
github.com/dlclark/regexp2 v1.4.0 // indirect
github.com/dustin/go-humanize v1.0.0
github.com/editorconfig/editorconfig-core-go/v2 v2.4.1
@@ -46,13 +45,14 @@ require (
github.com/go-openapi/errors v0.20.0 // indirect
github.com/go-openapi/validate v0.20.2 // indirect
github.com/go-redis/redis/v8 v8.6.0
github.com/go-sql-driver/mysql v1.5.0
github.com/go-sql-driver/mysql v1.6.0
github.com/go-swagger/go-swagger v0.26.1
github.com/go-testfixtures/testfixtures/v3 v3.5.0
github.com/gobwas/glob v0.2.3
github.com/gogs/chardet v0.0.0-20191104214054-4b6791f73a28
github.com/gogs/cron v0.0.0-20171120032916-9f6c956d3e14
github.com/gogs/go-gogs-client v0.0.0-20210131175652-1d7215cd8d85
github.com/golang-jwt/jwt v3.2.1+incompatible
github.com/golang/snappy v0.0.3 // indirect
github.com/google/go-github/v32 v32.1.0
github.com/google/uuid v1.2.0
@@ -67,21 +67,21 @@ require (
github.com/issue9/assert v1.3.2 // indirect
github.com/issue9/identicon v1.0.1
github.com/jaytaylor/html2text v0.0.0-20200412013138-3577fbdbcff7
github.com/json-iterator/go v1.1.10
github.com/json-iterator/go v1.1.11
github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51
github.com/kevinburke/ssh_config v0.0.0-20201106050909-4977a11b4351 // indirect
github.com/keybase/go-crypto v0.0.0-20200123153347-de78d2cb44f4
github.com/klauspost/compress v1.11.8
github.com/klauspost/pgzip v1.2.5 // indirect
github.com/lafriks/xormstore v1.4.0
github.com/lib/pq v1.9.0
github.com/lib/pq v1.10.2
github.com/libdns/libdns v0.2.0 // indirect
github.com/lunny/dingtalk_webhook v0.0.0-20171025031554-e3534c89ef96
github.com/mailru/easyjson v0.7.7 // indirect
github.com/markbates/goth v1.67.1
github.com/mattn/go-isatty v0.0.12
github.com/mattn/go-runewidth v0.0.10 // indirect
github.com/mattn/go-sqlite3 v1.14.6
github.com/markbates/goth v1.68.0
github.com/mattn/go-isatty v0.0.13
github.com/mattn/go-runewidth v0.0.13 // indirect
github.com/mattn/go-sqlite3 v1.14.8
github.com/mgechev/dots v0.0.0-20190921121421-c36f7dcfbb81
github.com/mgechev/revive v1.0.3
github.com/mholt/acmez v0.1.3 // indirect
@@ -107,7 +107,6 @@ require (
github.com/prometheus/common v0.18.0 // indirect
github.com/prometheus/procfs v0.6.0 // indirect
github.com/quasoft/websspi v1.0.0
github.com/rivo/uniseg v0.2.0 // indirect
github.com/russross/blackfriday/v2 v2.1.0 // indirect
github.com/sergi/go-diff v1.1.0
github.com/shurcooL/httpfs v0.0.0-20190707220628-8d4bc4ba7749 // indirect
@@ -135,10 +134,10 @@ require (
go.jolheiser.com/pwn v0.0.3
go.uber.org/multierr v1.6.0 // indirect
go.uber.org/zap v1.16.0 // indirect
golang.org/x/crypto v0.0.0-20210220033148-5ea612d1eb83
golang.org/x/crypto v0.0.0-20210616213533-5ff15b29337e
golang.org/x/net v0.0.0-20210614182718-04defd469f4e
golang.org/x/oauth2 v0.0.0-20210220000619-9bb904979d93
golang.org/x/sys v0.0.0-20210423082822-04245dca01da
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1
golang.org/x/text v0.3.6
golang.org/x/time v0.0.0-20210220033141-f8bda1e9f3ba // indirect
golang.org/x/tools v0.1.0
@@ -149,7 +148,9 @@ require (
mvdan.cc/xurls/v2 v2.2.0
strk.kbt.io/projects/go/libravatar v0.0.0-20191008002943-06d1c002b251
xorm.io/builder v0.3.9
xorm.io/xorm v1.1.0
xorm.io/xorm v1.2.2
)
replace github.com/hashicorp/go-version => github.com/6543/go-version v1.2.4
replace github.com/golang-jwt/jwt v3.2.1+incompatible => github.com/zeripath/jwt v3.2.2-go1.14+incompatible

123
go.sum
View File

@@ -63,6 +63,8 @@ github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym
github.com/GeertJohan/go.incremental v1.0.0/go.mod h1:6fAjUhbVuX1KcMD3c8TEgVUqmo4seqhv0i0kdATSkM0=
github.com/GeertJohan/go.rice v1.0.0/go.mod h1:eH6gbSOAUv07dQuZVnBmoDP8mgsM1rtixis4Tib9if0=
github.com/Knetic/govaluate v3.0.1-0.20171022003610-9aa49832a739+incompatible/go.mod h1:r7JcOSlj0wfOMncg0iLm8Leh48TZaKVeNIfJntJ2wa0=
github.com/Masterminds/semver/v3 v3.1.1 h1:hLg3sBzpNErnxhQtUy/mmLR2I9foDujNK030IGemrRc=
github.com/Masterminds/semver/v3 v3.1.1/go.mod h1:VPu/7SZ7ePZ3QOrcuXROw5FAcLl4a0cBrbBpGY/8hQs=
github.com/Microsoft/go-winio v0.4.14/go.mod h1:qXqCSQ3Xa7+6tgxaGTIe4Kpcdsi+P8jBhyzoq1bpyYA=
github.com/Microsoft/go-winio v0.4.16 h1:FtSW/jqD+l4ba5iPBj9CODVtgfYAD8w2wS923g/cFDk=
github.com/Microsoft/go-winio v0.4.16/go.mod h1:XB6nPKklQyQ7GC9LdcBEcBl8PF76WugXOPRXwdLnMv0=
@@ -249,9 +251,8 @@ github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/denisenkom/go-mssqldb v0.0.0-20191128021309-1d7a30a10f73/go.mod h1:xbL0rPBG9cCiLr28tMa8zpbdarY27NDyej4t/EjAShU=
github.com/denisenkom/go-mssqldb v0.0.0-20200428022330-06a60b6afbbc/go.mod h1:xbL0rPBG9cCiLr28tMa8zpbdarY27NDyej4t/EjAShU=
github.com/denisenkom/go-mssqldb v0.9.0 h1:RSohk2RsiZqLZ0zCjtfn3S4Gp4exhpBWHyQ7D0yGjAk=
github.com/denisenkom/go-mssqldb v0.9.0/go.mod h1:xbL0rPBG9cCiLr28tMa8zpbdarY27NDyej4t/EjAShU=
github.com/dgrijalva/jwt-go v3.2.0+incompatible h1:7qlOGliEKZXTDg6OTjfoBKDXWrumCAMpl/TFQ4/5kLM=
github.com/denisenkom/go-mssqldb v0.10.0 h1:QykgLZBorFE95+gO3u9esLd0BmbvpWp0/waNNZfHBM8=
github.com/denisenkom/go-mssqldb v0.10.0/go.mod h1:xbL0rPBG9cCiLr28tMa8zpbdarY27NDyej4t/EjAShU=
github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ=
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f h1:lO4WD4F/rVNCu3HqELle0jiPLLBs70cWOduZpkS1E78=
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc=
@@ -448,8 +449,9 @@ github.com/go-redis/redis/v8 v8.6.0 h1:swqbqOrxaPztsj2Hf1p94M3YAgl7hYEpcw21z299h
github.com/go-redis/redis/v8 v8.6.0/go.mod h1:DQ9q4Rk2HtwkrwVrdgmphoOQDMfpvcd/nHEwRsicg8s=
github.com/go-sql-driver/mysql v1.4.0/go.mod h1:zAC/RDZ24gD3HViQzih4MyKcchzm+sOG5ZlKdlhCg5w=
github.com/go-sql-driver/mysql v1.4.1/go.mod h1:zAC/RDZ24gD3HViQzih4MyKcchzm+sOG5ZlKdlhCg5w=
github.com/go-sql-driver/mysql v1.5.0 h1:ozyZYNQW3x3HtqT1jira07DN2PArx2v7/mN66gGcHOs=
github.com/go-sql-driver/mysql v1.5.0/go.mod h1:DCzpHaOWr8IXmIStZouvnhqoel9Qv2LBy8hT2VhHyBg=
github.com/go-sql-driver/mysql v1.6.0 h1:BCTh4TKNUYmOmMUcQ3IipzF5prigylS7XXjEkfCHuOE=
github.com/go-sql-driver/mysql v1.6.0/go.mod h1:DCzpHaOWr8IXmIStZouvnhqoel9Qv2LBy8hT2VhHyBg=
github.com/go-stack/stack v1.8.0 h1:5SgMzNM5HxrEjV0ww2lTmX6E2Izsfxas4+YHWRs3Lsk=
github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY=
github.com/go-swagger/go-swagger v0.26.1 h1:1XUWLnH6hKxHzeKjJfA2gHkSqcT1Zgi4q/PZp2hDdN8=
@@ -484,8 +486,11 @@ github.com/gobuffalo/packr/v2 v2.2.0/go.mod h1:CaAwI0GPIAv+5wKLtv8Afwl+Cm78K/I/V
github.com/gobuffalo/syncx v0.0.0-20190224160051-33c29581e754/go.mod h1:HhnNqWY95UYwwW3uSASeV7vtgYkT2t16hJgV3AEPUpw=
github.com/gobwas/glob v0.2.3 h1:A4xDbljILXROh+kObIiy5kIaPYD8e96x1tgBhUI5J+Y=
github.com/gobwas/glob v0.2.3/go.mod h1:d3Ez4x06l9bZtSvzIay5+Yzi0fmZzPgnTbPcKjJAkT8=
github.com/gofrs/uuid v3.2.0+incompatible h1:y12jRkkFxsd7GpqdSZ+/KCs/fJbqpEXSGd4+jfEaewE=
github.com/goccy/go-json v0.7.4 h1:B44qRUFwz/vxPKPISQ1KhvzRi9kZ28RAf6YtjriBZ5k=
github.com/goccy/go-json v0.7.4/go.mod h1:6MelG93GURQebXPDq3khkgXZkazVtN9CRI+MGFi0w8I=
github.com/gofrs/uuid v3.2.0+incompatible/go.mod h1:b2aQJv3Z4Fp6yNu3cdSllBxTCLRxnplIgP/c0N/04lM=
github.com/gofrs/uuid v4.0.0+incompatible h1:1SD/1F5pU8p29ybwgQSwpQk+mwdRrXCYuPhW6m+TnJw=
github.com/gofrs/uuid v4.0.0+incompatible/go.mod h1:b2aQJv3Z4Fp6yNu3cdSllBxTCLRxnplIgP/c0N/04lM=
github.com/gogo/googleapis v1.1.0/go.mod h1:gf4bu3Q80BeJ6H1S1vYPm8/ELATdvryBaNFGgqEef3s=
github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
github.com/gogo/protobuf v1.2.0/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
@@ -656,12 +661,18 @@ github.com/jackc/chunkreader/v2 v2.0.1/go.mod h1:odVSm741yZoC3dpHEUXIqA9tQRhFrgO
github.com/jackc/pgconn v0.0.0-20190420214824-7e0022ef6ba3/go.mod h1:jkELnwuX+w9qN5YIfX0fl88Ehu4XC3keFuOJJk9pcnA=
github.com/jackc/pgconn v0.0.0-20190824142844-760dd75542eb/go.mod h1:lLjNuW/+OfW9/pnVKPazfWOgNfH2aPem8YQ7ilXGvJE=
github.com/jackc/pgconn v0.0.0-20190831204454-2fabfa3c18b7/go.mod h1:ZJKsE/KZfsUgOEh9hBm+xYTstcNHg7UPMVJqRfQxq4s=
github.com/jackc/pgconn v1.5.0 h1:oFSOilzIZkyg787M1fEmyMfOUUvwj0daqYMfaWwNL4o=
github.com/jackc/pgconn v1.4.0/go.mod h1:Y2O3ZDF0q4mMacyWV3AstPJpeHXWGEetiFttmq5lahk=
github.com/jackc/pgconn v1.5.0/go.mod h1:QeD3lBfpTFe8WUnPZWN5KY/mB8FGMIYRdd8P8Jr0fAI=
github.com/jackc/pgconn v1.5.1-0.20200601181101-fa742c524853/go.mod h1:QeD3lBfpTFe8WUnPZWN5KY/mB8FGMIYRdd8P8Jr0fAI=
github.com/jackc/pgconn v1.8.0/go.mod h1:1C2Pb36bGIP9QHGBYCjnyhqu7Rv3sGshaQUvmfGIB/o=
github.com/jackc/pgconn v1.8.1/go.mod h1:JV6m6b6jhjdmzchES0drzCcYcAHS1OPD5xu3OZ/lE2g=
github.com/jackc/pgconn v1.9.0 h1:gqibKSTJup/ahCsNKyMZAniPuZEfIqfXFc8FOWVYR+Q=
github.com/jackc/pgconn v1.9.0/go.mod h1:YctiPyvzfU11JFxoXokUOOKQXQmDMoJL9vJzHH8/2JY=
github.com/jackc/pgio v1.0.0 h1:g12B9UwVnzGhueNavwioyEEpAmqMe1E/BN9ES+8ovkE=
github.com/jackc/pgio v1.0.0/go.mod h1:oP+2QK2wFfUWgr+gxjoBH9KGBb31Eio69xUb0w5bYf8=
github.com/jackc/pgmock v0.0.0-20190831213851-13a1b77aafa2 h1:JVX6jT/XfzNqIjye4717ITLaNwV9mWbJx0dLCpcRzdA=
github.com/jackc/pgmock v0.0.0-20190831213851-13a1b77aafa2/go.mod h1:fGZlG77KXmcq05nJLRkk0+p82V8B8Dw8KN2/V9c/OAE=
github.com/jackc/pgmock v0.0.0-20201204152224-4fe30f7445fd h1:eDErF6V/JPJON/B7s68BxwHgfmyOntHJQ8IOaz0x4R8=
github.com/jackc/pgmock v0.0.0-20201204152224-4fe30f7445fd/go.mod h1:hrBW0Enj2AZTNpt/7Y5rr2xe/9Mn757Wtb2xeBzPv2c=
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
github.com/jackc/pgproto3 v1.1.0 h1:FYYE4yRw+AgI8wXIinMlNjBbp/UitDJwfj5LqqewP1A=
@@ -670,25 +681,40 @@ github.com/jackc/pgproto3/v2 v2.0.0-alpha1.0.20190420180111-c116219b62db/go.mod
github.com/jackc/pgproto3/v2 v2.0.0-alpha1.0.20190609003834-432c2951c711/go.mod h1:uH0AWtUmuShn0bcesswc4aBTWGvw0cAxIJp+6OB//Wg=
github.com/jackc/pgproto3/v2 v2.0.0-rc3/go.mod h1:ryONWYqW6dqSg1Lw6vXNMXoBJhpzvWKnT95C46ckYeM=
github.com/jackc/pgproto3/v2 v2.0.0-rc3.0.20190831210041-4c03ce451f29/go.mod h1:ryONWYqW6dqSg1Lw6vXNMXoBJhpzvWKnT95C46ckYeM=
github.com/jackc/pgproto3/v2 v2.0.1 h1:Rdjp4NFjwHnEslx2b66FfCI2S0LhO4itac3hXz6WX9M=
github.com/jackc/pgproto3/v2 v2.0.1/go.mod h1:WfJCnwN3HIg9Ish/j3sgWXnAfK8A9Y0bwXYU5xKaEdA=
github.com/jackc/pgservicefile v0.0.0-20200307190119-3430c5407db8 h1:Q3tB+ExeflWUW7AFcAhXqk40s9mnNYLk1nOkKNZ5GnU=
github.com/jackc/pgproto3/v2 v2.0.6/go.mod h1:WfJCnwN3HIg9Ish/j3sgWXnAfK8A9Y0bwXYU5xKaEdA=
github.com/jackc/pgproto3/v2 v2.1.1 h1:7PQ/4gLoqnl87ZxL7xjO0DR5gYuviDCZxQJsUlFW1eI=
github.com/jackc/pgproto3/v2 v2.1.1/go.mod h1:WfJCnwN3HIg9Ish/j3sgWXnAfK8A9Y0bwXYU5xKaEdA=
github.com/jackc/pgservicefile v0.0.0-20200307190119-3430c5407db8/go.mod h1:vsD4gTJCa9TptPL8sPkXrLZ+hDuNrZCnj29CQpr4X1E=
github.com/jackc/pgservicefile v0.0.0-20200714003250-2b9c44734f2b h1:C8S2+VttkHFdOOCXJe+YGfa4vHYwlt4Zx+IVXQ97jYg=
github.com/jackc/pgservicefile v0.0.0-20200714003250-2b9c44734f2b/go.mod h1:vsD4gTJCa9TptPL8sPkXrLZ+hDuNrZCnj29CQpr4X1E=
github.com/jackc/pgtype v0.0.0-20190421001408-4ed0de4755e0/go.mod h1:hdSHsc1V01CGwFsrv11mJRHWJ6aifDLfdV3aVjFF0zg=
github.com/jackc/pgtype v0.0.0-20190824184912-ab885b375b90/go.mod h1:KcahbBH1nCMSo2DXpzsoWOAfFkdEtEJpPbVLq8eE+mc=
github.com/jackc/pgtype v0.0.0-20190828014616-a8802b16cc59/go.mod h1:MWlu30kVJrUS8lot6TQqcg7mtthZ9T0EoIBFiJcmcyw=
github.com/jackc/pgtype v1.3.0 h1:l8JvKrby3RI7Kg3bYEeU9TA4vqC38QDpFCfcrC7KuN0=
github.com/jackc/pgtype v1.2.0/go.mod h1:5m2OfMh1wTK7x+Fk952IDmI4nw3nPrvtQdM0ZT4WpC0=
github.com/jackc/pgtype v1.3.0/go.mod h1:b0JqxHvPmljG+HQ5IsvQ0yqeSi4nGcDTVjFoiLDb0Ik=
github.com/jackc/pgtype v1.3.1-0.20200510190516-8cd94a14c75a/go.mod h1:vaogEUkALtxZMCH411K+tKzNpwzCKU+AnPzBKZ+I+Po=
github.com/jackc/pgtype v1.3.1-0.20200606141011-f6355165a91c/go.mod h1:cvk9Bgu/VzJ9/lxTO5R5sf80p0DiucVtN7ZxvaC4GmQ=
github.com/jackc/pgtype v1.7.0/go.mod h1:ZnHF+rMePVqDKaOfJVI4Q8IVvAQMryDlDkZnKOI75BE=
github.com/jackc/pgtype v1.8.0 h1:iFVCcVhYlw0PulYCVoguRGm0SE9guIcPcccnLzHj8bA=
github.com/jackc/pgtype v1.8.0/go.mod h1:PqDKcEBtllAtk/2p6z6SHdXW5UB+MhE75tUol2OKexE=
github.com/jackc/pgx v3.6.2+incompatible h1:2zP5OD7kiyR3xzRYMhOcXVvkDZsImVXfj+yIyTQf3/o=
github.com/jackc/pgx v3.6.2+incompatible/go.mod h1:0ZGrqGqkRlliWnWB4zKnWtjbSWbGkVEFm4TeybAXq+I=
github.com/jackc/pgx/v4 v4.0.0-20190420224344-cc3461e65d96/go.mod h1:mdxmSJJuR08CZQyj1PVQBHy9XOp5p8/SHH6a0psbY9Y=
github.com/jackc/pgx/v4 v4.0.0-20190421002000-1b8f0016e912/go.mod h1:no/Y67Jkk/9WuGR0JG/JseM9irFbnEPbuWV2EELPNuM=
github.com/jackc/pgx/v4 v4.0.0-pre1.0.20190824185557-6972a5742186/go.mod h1:X+GQnOEnf1dqHGpw7JmHqHc1NxDoalibchSk9/RWuDc=
github.com/jackc/pgx/v4 v4.6.0 h1:Fh0O9GdlG4gYpjpwOqjdEodJUQM9jzN3Hdv7PN0xmm0=
github.com/jackc/pgx/v4 v4.5.0/go.mod h1:EpAKPLdnTorwmPUUsqrPxy5fphV18j9q3wrfRXgo+kA=
github.com/jackc/pgx/v4 v4.6.0/go.mod h1:vPh43ZzxijXUVJ+t/EmXBtFmbFVO72cuneCT9oAlxAg=
github.com/jackc/pgx/v4 v4.6.1-0.20200510190926-94ba730bb1e9/go.mod h1:t3/cdRQl6fOLDxqtlyhe9UWgfIi9R8+8v8GKV5TRA/o=
github.com/jackc/pgx/v4 v4.6.1-0.20200606145419-4e5062306904/go.mod h1:ZDaNWkt9sW1JMiNn0kdYBaLelIhw7Pg4qd+Vk6tw7Hg=
github.com/jackc/pgx/v4 v4.11.0/go.mod h1:i62xJgdrtVDsnL3U8ekyrQXEwGNTRoG7/8r+CIdYfcc=
github.com/jackc/pgx/v4 v4.12.0 h1:xiP3TdnkwyslWNp77yE5XAPfxAsU9RMFDe0c1SwN8h4=
github.com/jackc/pgx/v4 v4.12.0/go.mod h1:fE547h6VulLPA3kySjfnSG/e2D861g/50JlVUa/ub60=
github.com/jackc/puddle v0.0.0-20190413234325-e4ced69a3a2b/go.mod h1:m4B5Dj62Y0fbyuIc15OsIqK0+JU8nkqQjsgx7dvjSWk=
github.com/jackc/puddle v0.0.0-20190608224051-11cab39313c9/go.mod h1:m4B5Dj62Y0fbyuIc15OsIqK0+JU8nkqQjsgx7dvjSWk=
github.com/jackc/puddle v1.1.0/go.mod h1:m4B5Dj62Y0fbyuIc15OsIqK0+JU8nkqQjsgx7dvjSWk=
github.com/jackc/puddle v1.1.1/go.mod h1:m4B5Dj62Y0fbyuIc15OsIqK0+JU8nkqQjsgx7dvjSWk=
github.com/jackc/puddle v1.1.3/go.mod h1:m4B5Dj62Y0fbyuIc15OsIqK0+JU8nkqQjsgx7dvjSWk=
github.com/jarcoal/httpmock v0.0.0-20180424175123-9c70cfe4a1da/go.mod h1:ks+b9deReOc7jgqp+e7LuFiCBH6Rm5hL32cLcEAArb4=
github.com/jaytaylor/html2text v0.0.0-20200412013138-3577fbdbcff7 h1:g0fAGBisHaEQ0TRq1iBvemFRf+8AEWEmBESSiWB3Vsc=
github.com/jaytaylor/html2text v0.0.0-20200412013138-3577fbdbcff7/go.mod h1:CVKlgaMiht+LXvHG173ujK6JUhZXKb2u/BQtjPDIvyk=
@@ -708,8 +734,9 @@ github.com/jpillora/backoff v1.0.0/go.mod h1:J/6gKK9jxlEcS3zixgDgUAsiuZ7yrSoa/FX
github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU=
github.com/json-iterator/go v1.1.7/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
github.com/json-iterator/go v1.1.8/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
github.com/json-iterator/go v1.1.10 h1:Kz6Cvnvv2wGdaG/V8yMvfkmNiXq9Ya2KUv4rouJJr68=
github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
github.com/json-iterator/go v1.1.11 h1:uVUAXhF2To8cbw/3xN3pxj6kk7TYKs98NIrTqPlMWAQ=
github.com/json-iterator/go v1.1.11/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU=
github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk=
github.com/jtolds/gls v4.2.1+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
@@ -768,8 +795,8 @@ github.com/lib/pq v1.1.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
github.com/lib/pq v1.2.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
github.com/lib/pq v1.3.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
github.com/lib/pq v1.7.0/go.mod h1:AlVN5x4E4T544tWzH6hKfbfQvm3HdbOxrmggDNAPY9o=
github.com/lib/pq v1.9.0 h1:L8nSXQQzAYByakOFMTwpjRoHsMJklur4Gi59b6VivR8=
github.com/lib/pq v1.9.0/go.mod h1:AlVN5x4E4T544tWzH6hKfbfQvm3HdbOxrmggDNAPY9o=
github.com/lib/pq v1.10.2 h1:AqzbZs4ZoCBp+GtejcpCpcxM3zlSMx29dXbUSeVtJb8=
github.com/lib/pq v1.10.2/go.mod h1:AlVN5x4E4T544tWzH6hKfbfQvm3HdbOxrmggDNAPY9o=
github.com/libdns/libdns v0.1.0/go.mod h1:yQCXzk1lEZmmCPa857bnk4TsOiqYasqpyOEeSObbb40=
github.com/libdns/libdns v0.2.0 h1:ewg3ByWrdUrxrje8ChPVMBNcotg7H9LQYg+u5De2RzI=
github.com/libdns/libdns v0.2.0/go.mod h1:yQCXzk1lEZmmCPa857bnk4TsOiqYasqpyOEeSObbb40=
@@ -793,8 +820,8 @@ github.com/mailru/easyjson v0.7.6/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJ
github.com/mailru/easyjson v0.7.7 h1:UGYAvKxe3sBsEDzO8ZeWOSlIQfWFlxbzLZe7hwFURr0=
github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc=
github.com/markbates/going v1.0.0/go.mod h1:I6mnB4BPnEeqo85ynXIx1ZFLLbtiLHNXVgWeFO9OGOA=
github.com/markbates/goth v1.67.1 h1:gU5B0pzHVyhnJPwGynfFnkfvaQ39C1Sy+ewdl+bhAOw=
github.com/markbates/goth v1.67.1/go.mod h1:EyLFHGU5ySr2GXRDyJH5nu2dA7parbC8QwIYW/rGcWg=
github.com/markbates/goth v1.68.0 h1:90sKvjRAKHcl9V2uC9x/PJXeD78cFPiBsyP1xVhoQfA=
github.com/markbates/goth v1.68.0/go.mod h1:V2VcDMzDiMHW+YmqYl7i0cMiAUeCkAe4QE6jRKBhXZw=
github.com/markbates/oncer v0.0.0-20181203154359-bf2de49a0be2/go.mod h1:Ld9puTsIW75CHf65OeIOkyKbteujpZVXDpWK6YGZbxE=
github.com/markbates/safe v1.0.1/go.mod h1:nAqgmRi7cY2nqMc92/bSEeQA+R4OheNU2T1kNSCBdG0=
github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU=
@@ -809,17 +836,19 @@ github.com/mattn/go-isatty v0.0.5/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hd
github.com/mattn/go-isatty v0.0.7/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s=
github.com/mattn/go-isatty v0.0.8/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s=
github.com/mattn/go-isatty v0.0.9/go.mod h1:YNRxwqDuOph6SZLI9vUUz6OYw3QyUt7WiY2yME+cCiQ=
github.com/mattn/go-isatty v0.0.12 h1:wuysRhFDzyxgEmMf5xjvJ2M9dZoWAXNNr5LSBS7uHXY=
github.com/mattn/go-isatty v0.0.12/go.mod h1:cbi8OIDigv2wuxKPP5vlRcQ1OAZbq2CE4Kysco4FUpU=
github.com/mattn/go-isatty v0.0.13 h1:qdl+GuBjcsKKDco5BsxPJlId98mSWNKqYA+Co0SC1yA=
github.com/mattn/go-isatty v0.0.13/go.mod h1:cbi8OIDigv2wuxKPP5vlRcQ1OAZbq2CE4Kysco4FUpU=
github.com/mattn/go-runewidth v0.0.2/go.mod h1:LwmH8dsx7+W8Uxz3IHJYH5QSwggIsqBzpuz5H//U1FU=
github.com/mattn/go-runewidth v0.0.7/go.mod h1:H031xJmbD/WCDINGzjvQ9THkh0rPKHF+m2gUSrubnMI=
github.com/mattn/go-runewidth v0.0.9/go.mod h1:H031xJmbD/WCDINGzjvQ9THkh0rPKHF+m2gUSrubnMI=
github.com/mattn/go-runewidth v0.0.10 h1:CoZ3S2P7pvtP45xOtBw+/mDL2z0RKI576gSkzRRpdGg=
github.com/mattn/go-runewidth v0.0.10/go.mod h1:RAqKPSqVFrSLVXbA8x7dzmKdmGzieGRCM46jaSJTDAk=
github.com/mattn/go-runewidth v0.0.13 h1:lTGmDsbAYt5DmK6OnoV7EuIF1wEIFAcxld6ypU4OSgU=
github.com/mattn/go-runewidth v0.0.13/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh153qUoGf23w=
github.com/mattn/go-sqlite3 v1.11.0/go.mod h1:FPy6KqzDD04eiIsT53CuJW3U88zkxoIYsOqkbpncsNc=
github.com/mattn/go-sqlite3 v1.14.0/go.mod h1:JIl7NbARA7phWnGvh0LKTyg7S9BA+6gx71ShQilpsus=
github.com/mattn/go-sqlite3 v1.14.6 h1:dNPt6NO46WmLVt2DLNpwczCmdV5boIZ6g/tlDrlRUbg=
github.com/mattn/go-sqlite3 v1.14.6/go.mod h1:NyWgC/yNuGj7Q9rpYnZvas74GogHl5/Z4A/KQRfk6bU=
github.com/mattn/go-sqlite3 v1.14.8 h1:gDp86IdQsN/xWjIEmr9MF6o9mpksUgh0fu+9ByFxzIU=
github.com/mattn/go-sqlite3 v1.14.8/go.mod h1:NyWgC/yNuGj7Q9rpYnZvas74GogHl5/Z4A/KQRfk6bU=
github.com/matttproud/golang_protobuf_extensions v1.0.1 h1:4hp9jkHxhMHkqkrB3Ix0jegS5sx/RkqARlsWZ6pIwiU=
github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
github.com/mgechev/dots v0.0.0-20190921121421-c36f7dcfbb81 h1:QASJXOGm2RZ5Ardbc86qNFvby9AqkLDibfChMtAg5QM=
@@ -999,7 +1028,6 @@ github.com/rcrowley/go-metrics v0.0.0-20181016184325-3113b8401b8a/go.mod h1:bCqn
github.com/rcrowley/go-metrics v0.0.0-20190826022208-cac0b30c2563/go.mod h1:bCqnVzQkZxMG4s8nGwiZ5l3QUCyqpo9Y+/ZMZ9VjZe4=
github.com/remyoudompheng/bigfft v0.0.0-20200410134404-eec4a21b6bb0 h1:OdAsTTz6OkFY5QxjkYwrChwuRruF69c169dPK26NUlk=
github.com/remyoudompheng/bigfft v0.0.0-20200410134404-eec4a21b6bb0/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
github.com/rivo/uniseg v0.1.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
github.com/rivo/uniseg v0.2.0 h1:S1pD9weZBuJdFmowNwbpi7BJ8TNftyUImj/0WQi72jY=
github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af/go.mod h1:XWv6SoW27p1b0cqNHllgS5HIMJraePCO15w5zCzIWYg=
@@ -1023,8 +1051,10 @@ github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529/go.mod h1:DxrIzT+xaE7yg
github.com/sergi/go-diff v1.0.0/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAmXWZgo=
github.com/sergi/go-diff v1.1.0 h1:we8PVUC3FE2uYfodKH/nBHMSetSfHDR6scGdBi+erh0=
github.com/sergi/go-diff v1.1.0/go.mod h1:STckp+ISIX8hZLjrqAeVduY0gWCT9IjLuqbuNXdaHfM=
github.com/shopspring/decimal v0.0.0-20180709203117-cd690d0c9e24 h1:pntxY8Ary0t43dCZ5dqY4YTJCObLY1kIXl0uzMv+7DE=
github.com/shopspring/decimal v0.0.0-20180709203117-cd690d0c9e24/go.mod h1:M+9NzErvs504Cn4c5DxATwIqPbtswREoFCre64PpcG4=
github.com/shopspring/decimal v0.0.0-20200227202807-02e2044944cc/go.mod h1:DKyhrW/HYNuLGql+MJL6WCR6knT2jwCFRcu2hWCYk4o=
github.com/shopspring/decimal v1.2.0 h1:abSATXmQEYyShuxI4/vyW3tV1MrKAJzCZ/0zLUXYbsQ=
github.com/shopspring/decimal v1.2.0/go.mod h1:DKyhrW/HYNuLGql+MJL6WCR6knT2jwCFRcu2hWCYk4o=
github.com/shurcooL/httpfs v0.0.0-20190707220628-8d4bc4ba7749 h1:bUGsEnyNbVPw06Bs80sCeARAlK8lhwqGyi6UT8ymuGk=
github.com/shurcooL/httpfs v0.0.0-20190707220628-8d4bc4ba7749/go.mod h1:ZY1cvUeJuFPAdZ/B6v7RHavJWZn2YPVFQ1OSXhCGOkg=
github.com/shurcooL/sanitized_anchor_name v1.0.0/go.mod h1:1NzhyTcUVG4SuEtjjoZeVRXNmyL/1OwPU0+IJeTBvfc=
@@ -1153,6 +1183,8 @@ github.com/yuin/goldmark-highlighting v0.0.0-20200307114337-60d527fdb691/go.mod
github.com/yuin/goldmark-meta v1.0.0 h1:ScsatUIT2gFS6azqzLGUjgOnELsBOxMXerM3ogdJhAM=
github.com/yuin/goldmark-meta v1.0.0/go.mod h1:zsNNOrZ4nLuyHAJeLQEZcQat8dm70SmB2kHbls092Gc=
github.com/zenazn/goji v0.9.0/go.mod h1:7S9M489iMyHBNxwZnk9/EHS098H4/F6TATF2mIxtB1Q=
github.com/zeripath/jwt v3.2.2-go1.14+incompatible h1:jqxA3KuCQRLn0lHdt1G8t1EUJ92FmRUFnXHghVvJLJs=
github.com/zeripath/jwt v3.2.2-go1.14+incompatible/go.mod h1:pYPrRXN84mQC6u5c/08icdKllASIBEOurvsTPbDurLs=
github.com/ziutek/mymysql v1.5.4/go.mod h1:LMSpPZ6DbqWFxNCHW77HeMg9I646SAhApZ/wKdgO/C0=
go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
go.etcd.io/bbolt v1.3.3/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
@@ -1223,6 +1255,7 @@ golang.org/x/crypto v0.0.0-20190611184440-5c40567a22f8/go.mod h1:yigFU9vqHzYiE8U
golang.org/x/crypto v0.0.0-20190617133340-57b3e21c3d56/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20190701094942-4def268fd1a4/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20190820162420-60c769a6c586/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20190911031432-227b76d455e7/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20190927123631-a832865fa7ad/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20200302210943-78000ba7a073/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
@@ -1232,8 +1265,10 @@ golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPh
golang.org/x/crypto v0.0.0-20200709230013-948cd5f35899/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20200728195943-123391ffb6de/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20201124201722-c8d3bf9c5392/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I=
golang.org/x/crypto v0.0.0-20210220033148-5ea612d1eb83 h1:/ZScEX8SfEmUGRHs0gxpqteO5nfNW6axyZbBdw9A12g=
golang.org/x/crypto v0.0.0-20210220033148-5ea612d1eb83/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I=
golang.org/x/crypto v0.0.0-20201203163018-be400aefbc4c/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I=
golang.org/x/crypto v0.0.0-20210322153248-0c34fe9e7dc2/go.mod h1:T9bdIzuCu7OtxOm1hfPfRQxPLYneinmdGuTeoZ9dtd4=
golang.org/x/crypto v0.0.0-20210616213533-5ff15b29337e h1:gsTQYXdTw2Gq7RBsWvlQ91b+aEQ6bXFUngBGuR8sPpI=
golang.org/x/crypto v0.0.0-20210616213533-5ff15b29337e/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
@@ -1322,6 +1357,7 @@ golang.org/x/net v0.0.0-20201202161906-c7110b5ffcbb/go.mod h1:sp8m0HH+o8qH0wwXwY
golang.org/x/net v0.0.0-20201209123823-ac852fbbde11/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20201224014010-6772e930b67b/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20210119194325-5f4716e94777/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20210614182718-04defd469f4e h1:XpT3nA5TvE525Ne3hInMh6+GETgn27Zfm9dxsThnX2Q=
golang.org/x/net v0.0.0-20210614182718-04defd469f4e/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
@@ -1419,8 +1455,9 @@ golang.org/x/sys v0.0.0-20210112080510-489259a85091/go.mod h1:h1NjWce9XRLGQEsW7w
golang.org/x/sys v0.0.0-20210119212857-b64e53b001e4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210124154548-22da62e12c0c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210220050731-9a76102bfb43/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210423082822-04245dca01da h1:b3NXsE2LusjYGGjL5bxEVZZORm/YEFFrWFjR8eFrw/c=
golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1 h1:SrN+KX8Art/Sf4HNj6Zcz06G7VEz+7w9tdXTPOZ7+l4=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1 h1:v+OssWQX+hTHEmOBgwxdZxK4zHq3yOs8F9J7mk0PY8E=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
@@ -1670,31 +1707,34 @@ honnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt
honnef.co/go/tools v0.0.1-2020.1.3/go.mod h1:X/FiERA/W4tHapMX5mGpAtMSVEeEUOyHaw9vFzvIQ3k=
honnef.co/go/tools v0.0.1-2020.1.4 h1:UoveltGrhghAA7ePc+e+QYDHXrBps2PqFZiHkGR/xK8=
honnef.co/go/tools v0.0.1-2020.1.4/go.mod h1:X/FiERA/W4tHapMX5mGpAtMSVEeEUOyHaw9vFzvIQ3k=
modernc.org/cc/v3 v3.31.5-0.20210308123301-7a3e9dab9009 h1:u0oCo5b9wyLr++HF3AN9JicGhkUxJhMz51+8TIZH9N0=
modernc.org/cc/v3 v3.31.5-0.20210308123301-7a3e9dab9009/go.mod h1:0R6jl1aZlIl2avnYfbfHBS1QB6/f+16mihBObaBC878=
modernc.org/ccgo/v3 v3.9.0 h1:JbcEIqjw4Agf+0g3Tc85YvfYqkkFOv6xBwS4zkfqSoA=
modernc.org/ccgo/v3 v3.9.0/go.mod h1:nQbgkn8mwzPdp4mm6BT6+p85ugQ7FrGgIcYaE7nSrpY=
lukechampine.com/uint128 v1.1.1 h1:pnxCASz787iMf+02ssImqk6OLt+Z5QHMoZyUXR4z6JU=
lukechampine.com/uint128 v1.1.1/go.mod h1:c4eWIwlEGaxC/+H1VguhU4PHXNWDCDMUlWdIWl2j1gk=
modernc.org/cc/v3 v3.33.6 h1:r63dgSzVzRxUpAJFPQWHy1QeZeY1ydNENUDaBx1GqYc=
modernc.org/cc/v3 v3.33.6/go.mod h1:iPJg1pkwXqAV16SNgFBVYmggfMg6xhs+2oiO0vclK3g=
modernc.org/ccgo/v3 v3.9.5 h1:dEuUSf8WN51rDkprFuAqjfchKEzN0WttP/Py3enBwjk=
modernc.org/ccgo/v3 v3.9.5/go.mod h1:umuo2EP2oDSBnD3ckjaVUXMrmeAw8C8OSICVa0iFf60=
modernc.org/httpfs v1.0.6 h1:AAgIpFZRXuYnkjftxTAZwMIiwEqAfk8aVB2/oA6nAeM=
modernc.org/httpfs v1.0.6/go.mod h1:7dosgurJGp0sPaRanU53W4xZYKh14wfzX420oZADeHM=
modernc.org/libc v1.7.13-0.20210308123627-12f642a52bb8/go.mod h1:U1eq8YWr/Kc1RWCMFUWEdkTg8OTcfLw2kY8EDwl039w=
modernc.org/libc v1.8.0 h1:Pp4uv9g0csgBMpGPABKtkieF6O5MGhfGo6ZiOdlYfR8=
modernc.org/libc v1.8.0/go.mod h1:U1eq8YWr/Kc1RWCMFUWEdkTg8OTcfLw2kY8EDwl039w=
modernc.org/libc v1.9.8/go.mod h1:U1eq8YWr/Kc1RWCMFUWEdkTg8OTcfLw2kY8EDwl039w=
modernc.org/libc v1.9.11 h1:QUxZMs48Ahg2F7SN41aERvMfGLY2HU/ADnB9DC4Yts8=
modernc.org/libc v1.9.11/go.mod h1:NyF3tsA5ArIjJ83XB0JlqhjTabTCHm9aX4XMPHyQn0Q=
modernc.org/mathutil v1.1.1/go.mod h1:mZW8CKdRPY1v87qxC/wUdX5O1qDzXMP5TH3wjfpga6E=
modernc.org/mathutil v1.2.2 h1:+yFk8hBprV+4c0U9GjFtL+dV3N8hOJ8JCituQcMShFY=
modernc.org/mathutil v1.2.2/go.mod h1:mZW8CKdRPY1v87qxC/wUdX5O1qDzXMP5TH3wjfpga6E=
modernc.org/mathutil v1.4.0 h1:GCjoRaBew8ECCKINQA2nYjzvufFW9YiEuuB+rQ9bn2E=
modernc.org/mathutil v1.4.0/go.mod h1:mZW8CKdRPY1v87qxC/wUdX5O1qDzXMP5TH3wjfpga6E=
modernc.org/memory v1.0.4 h1:utMBrFcpnQDdNsmM6asmyH/FM9TqLPS7XF7otpJmrwM=
modernc.org/memory v1.0.4/go.mod h1:nV2OApxradM3/OVbs2/0OsP6nPfakXpi50C7dcoHXlc=
modernc.org/opt v0.1.1 h1:/0RX92k9vwVeDXj+Xn23DKp2VJubL7k8qNffND6qn3A=
modernc.org/opt v0.1.1/go.mod h1:WdSiB5evDcignE70guQKxYUl14mgWtbClRi5wmkkTX0=
modernc.org/sqlite v1.10.1-0.20210314190707-798bbeb9bb84 h1:rgEUzE849tFlHSoeCrKyS9cZAljC+DY7MdMHKq6R6sY=
modernc.org/sqlite v1.10.1-0.20210314190707-798bbeb9bb84/go.mod h1:PGzq6qlhyYjL6uVbSgS6WoF7ZopTW/sI7+7p+mb4ZVU=
modernc.org/strutil v1.1.0 h1:+1/yCzZxY2pZwwrsbH+4T7BQMoLQ9QiBshRC9eicYsc=
modernc.org/strutil v1.1.0/go.mod h1:lstksw84oURvj9y3tn8lGvRxyRC1S2+g5uuIzNfIOBs=
modernc.org/tcl v1.5.0 h1:euZSUNfE0Fd4W8VqXI1Ly1v7fqDJoBuAV88Ea+SnaSs=
modernc.org/tcl v1.5.0/go.mod h1:gb57hj4pO8fRrK54zveIfFXBaMHK3SKJNWcmRw1cRzc=
modernc.org/sqlite v1.11.2 h1:ShWQpeD3ag/bmx6TqidBlIWonWmQaSQKls3aenCbt+w=
modernc.org/sqlite v1.11.2/go.mod h1:+mhs/P1ONd+6G7hcAs6irwDi/bjTQ7nLW6LHRBsEa3A=
modernc.org/strutil v1.1.1 h1:xv+J1BXY3Opl2ALrBwyfEikFAj8pmqcpnfmuwUwcozs=
modernc.org/strutil v1.1.1/go.mod h1:DE+MQQ/hjKBZS2zNInV5hhcipt5rLPWkmpbGeW5mmdw=
modernc.org/tcl v1.5.5 h1:N03RwthgTR/l/eQvz3UjfYnvVVj1G2sZqzFGfoD4HE4=
modernc.org/tcl v1.5.5/go.mod h1:ADkaTUuwukkrlhqwERyq0SM8OvyXo7+TjFz7yAF56EI=
modernc.org/token v1.0.0 h1:a0jaWiNMDhDUtqOj09wvjWWAqd3q7WpBulmL9H2egsk=
modernc.org/token v1.0.0/go.mod h1:UGzOrNV1mAFSEB63lOFHIpNRUVMvYTc6yu1SMY/XTDM=
modernc.org/z v1.0.1-0.20210308123920-1f282aa71362/go.mod h1:8/SRk5C/HgiQWCgXdfpb+1RvhORdkz5sw72d3jjtyqA=
modernc.org/z v1.0.1 h1:WyIDpEpAIx4Hel6q/Pcgj/VhaQV5XPJ2I6ryIYbjnpc=
modernc.org/z v1.0.1/go.mod h1:8/SRk5C/HgiQWCgXdfpb+1RvhORdkz5sw72d3jjtyqA=
mvdan.cc/xurls/v2 v2.2.0 h1:NSZPykBXJFCetGZykLAxaL6SIpvbVy/UFEniIfHAa8A=
@@ -1707,9 +1747,8 @@ sourcegraph.com/sourcegraph/appdash v0.0.0-20190731080439-ebfcffb1b5c0/go.mod h1
strk.kbt.io/projects/go/libravatar v0.0.0-20191008002943-06d1c002b251 h1:mUcz5b3FJbP5Cvdq7Khzn6J9OCUQJaBwgBkCR+MOwSs=
strk.kbt.io/projects/go/libravatar v0.0.0-20191008002943-06d1c002b251/go.mod h1:FJGmPh3vz9jSos1L/F91iAgnC/aejc0wIIrF2ZwJxdY=
xorm.io/builder v0.3.7/go.mod h1:aUW0S9eb9VCaPohFCH3j7czOx1PMW3i1HrSzbLYGBSE=
xorm.io/builder v0.3.8/go.mod h1:aUW0S9eb9VCaPohFCH3j7czOx1PMW3i1HrSzbLYGBSE=
xorm.io/builder v0.3.9 h1:Sd65/LdWyO7LR8+Cbd+e7mm3sK/7U9k0jS3999IDHMc=
xorm.io/builder v0.3.9/go.mod h1:aUW0S9eb9VCaPohFCH3j7czOx1PMW3i1HrSzbLYGBSE=
xorm.io/xorm v1.0.6/go.mod h1:uF9EtbhODq5kNWxMbnBEj8hRRZnlcNSz2t2N7HW/+A4=
xorm.io/xorm v1.1.0 h1:mkEsQXLauZajiOld2cB2PkFcUZKePepPgs1bC1dw8RA=
xorm.io/xorm v1.1.0/go.mod h1:EDzNHMuCVZNszkIRSLL2nI0zX+nQE8RstAVranlSfqI=
xorm.io/xorm v1.2.2 h1:FFBOEvJ++8fYBA9cywf2sxDVmFktl1SpJzTAG1ab06Y=
xorm.io/xorm v1.2.2/go.mod h1:fTG8tSjk6O1BYxwuohZUK+S1glnRycsCF05L1qQyEU0=

View File

@@ -144,6 +144,60 @@ func TestLDAPUserSignin(t *testing.T) {
assert.Equal(t, u.Email, htmlDoc.Find(`label[for="email"]`).Siblings().First().Text())
}
func TestLDAPAuthChange(t *testing.T) {
defer prepareTestEnv(t)()
addAuthSourceLDAP(t, "")
session := loginUser(t, "user1")
req := NewRequest(t, "GET", "/admin/auths")
resp := session.MakeRequest(t, req, http.StatusOK)
doc := NewHTMLParser(t, resp.Body)
href, exists := doc.Find("table.table td a").Attr("href")
if !exists {
assert.True(t, exists, "No authentication source found")
return
}
req = NewRequest(t, "GET", href)
resp = session.MakeRequest(t, req, http.StatusOK)
doc = NewHTMLParser(t, resp.Body)
csrf := doc.GetCSRF()
host, _ := doc.Find(`input[name="host"]`).Attr("value")
assert.Equal(t, host, getLDAPServerHost())
binddn, _ := doc.Find(`input[name="bind_dn"]`).Attr("value")
assert.Equal(t, binddn, "uid=gitea,ou=service,dc=planetexpress,dc=com")
req = NewRequestWithValues(t, "POST", href, map[string]string{
"_csrf": csrf,
"type": "2",
"name": "ldap",
"host": getLDAPServerHost(),
"port": "389",
"bind_dn": "uid=gitea,ou=service,dc=planetexpress,dc=com",
"bind_password": "password",
"user_base": "ou=people,dc=planetexpress,dc=com",
"filter": "(&(objectClass=inetOrgPerson)(memberOf=cn=git,ou=people,dc=planetexpress,dc=com)(uid=%s))",
"admin_filter": "(memberOf=cn=admin_staff,ou=people,dc=planetexpress,dc=com)",
"restricted_filter": "(uid=leela)",
"attribute_username": "uid",
"attribute_name": "givenName",
"attribute_surname": "sn",
"attribute_mail": "mail",
"attribute_ssh_public_key": "",
"is_sync_enabled": "on",
"is_active": "on",
})
session.MakeRequest(t, req, http.StatusFound)
req = NewRequest(t, "GET", href)
resp = session.MakeRequest(t, req, http.StatusOK)
doc = NewHTMLParser(t, resp.Body)
host, _ = doc.Find(`input[name="host"]`).Attr("value")
assert.Equal(t, host, getLDAPServerHost())
binddn, _ = doc.Find(`input[name="bind_dn"]`).Attr("value")
assert.Equal(t, binddn, "uid=gitea,ou=service,dc=planetexpress,dc=com")
}
func TestLDAPUserSync(t *testing.T) {
if skipLDAPTests() {
t.Skip()

View File

@@ -7,6 +7,7 @@ package models
import (
"crypto/tls"
"encoding/binary"
"errors"
"fmt"
"net/smtp"
@@ -69,11 +70,30 @@ var (
_ convert.Conversion = &SSPIConfig{}
)
// jsonUnmarshalIgnoreErroneousBOM - due to a bug in xorm (see https://gitea.com/xorm/xorm/pulls/1957) - it's
// possible that a Blob may gain an unwanted prefix of 0xff 0xfe.
func jsonUnmarshalIgnoreErroneousBOM(bs []byte, v interface{}) error {
// jsonUnmarshalHandleDoubleEncode - due to a bug in xorm (see https://gitea.com/xorm/xorm/pulls/1957) - it's
// possible that a Blob may be double encoded or gain an unwanted prefix of 0xff 0xfe.
func jsonUnmarshalHandleDoubleEncode(bs []byte, v interface{}) error {
json := jsoniter.ConfigCompatibleWithStandardLibrary
err := json.Unmarshal(bs, v)
if err != nil {
ok := true
rs := []byte{}
temp := make([]byte, 2)
for _, rn := range string(bs) {
if rn > 0xffff {
ok = false
break
}
binary.LittleEndian.PutUint16(temp, uint16(rn))
rs = append(rs, temp...)
}
if ok {
if rs[0] == 0xff && rs[1] == 0xfe {
rs = rs[2:]
}
err = json.Unmarshal(rs, v)
}
}
if err != nil && len(bs) > 2 && bs[0] == 0xff && bs[1] == 0xfe {
err = json.Unmarshal(bs[2:], v)
}
@@ -87,7 +107,7 @@ type LDAPConfig struct {
// FromDB fills up a LDAPConfig from serialized format.
func (cfg *LDAPConfig) FromDB(bs []byte) error {
return jsonUnmarshalIgnoreErroneousBOM(bs, &cfg)
return jsonUnmarshalHandleDoubleEncode(bs, &cfg)
}
// ToDB exports a LDAPConfig to a serialized format.
@@ -114,7 +134,7 @@ type SMTPConfig struct {
// FromDB fills up an SMTPConfig from serialized format.
func (cfg *SMTPConfig) FromDB(bs []byte) error {
return jsonUnmarshalIgnoreErroneousBOM(bs, cfg)
return jsonUnmarshalHandleDoubleEncode(bs, cfg)
}
// ToDB exports an SMTPConfig to a serialized format.
@@ -131,7 +151,7 @@ type PAMConfig struct {
// FromDB fills up a PAMConfig from serialized format.
func (cfg *PAMConfig) FromDB(bs []byte) error {
return jsonUnmarshalIgnoreErroneousBOM(bs, cfg)
return jsonUnmarshalHandleDoubleEncode(bs, cfg)
}
// ToDB exports a PAMConfig to a serialized format.
@@ -152,7 +172,7 @@ type OAuth2Config struct {
// FromDB fills up an OAuth2Config from serialized format.
func (cfg *OAuth2Config) FromDB(bs []byte) error {
return jsonUnmarshalIgnoreErroneousBOM(bs, cfg)
return jsonUnmarshalHandleDoubleEncode(bs, cfg)
}
// ToDB exports an SMTPConfig to a serialized format.
@@ -172,7 +192,7 @@ type SSPIConfig struct {
// FromDB fills up an SSPIConfig from serialized format.
func (cfg *SSPIConfig) FromDB(bs []byte) error {
return jsonUnmarshalIgnoreErroneousBOM(bs, cfg)
return jsonUnmarshalHandleDoubleEncode(bs, cfg)
}
// ToDB exports an SSPIConfig to a serialized format.

View File

@@ -565,11 +565,26 @@ func recreateTable(sess *xorm.Session, bean interface{}) error {
return err
}
if err := sess.Table(tempTableName).DropIndexes(bean); err != nil {
log.Error("Unable to drop indexes on temporary table %s. Error: %v", tempTableName, err)
return err
}
// SQLite and MySQL will move all the constraints from the temporary table to the new table
if _, err := sess.Exec(fmt.Sprintf("ALTER TABLE `%s` RENAME TO `%s`", tempTableName, tableName)); err != nil {
log.Error("Unable to rename %s to %s. Error: %v", tempTableName, tableName, err)
return err
}
if err := sess.Table(tableName).CreateIndexes(bean); err != nil {
log.Error("Unable to recreate indexes on table %s. Error: %v", tableName, err)
return err
}
if err := sess.Table(tableName).CreateUniques(bean); err != nil {
log.Error("Unable to recreate uniques on table %s. Error: %v", tableName, err)
return err
}
case setting.Database.UsePostgreSQL:
var originalSequences []string
type sequenceData struct {

View File

@@ -5,6 +5,8 @@
package migrations
import (
"context"
"xorm.io/xorm"
)
@@ -40,8 +42,17 @@ func convertTaskTypeToString(x *xorm.Engine) error {
return err
}
// to keep the migration could be rerun
exist, err := x.Dialect().IsColumnExist(x.DB(), context.Background(), "hook_task", "type")
if err != nil {
return err
}
if !exist {
return nil
}
for i, s := range hookTaskTypes {
if _, err := x.Exec("UPDATE hook_task set typ = ? where type=?", s, i); err != nil {
if _, err := x.Exec("UPDATE hook_task set typ = ? where `type`=?", s, i); err != nil {
return err
}
}

View File

@@ -33,7 +33,7 @@ type Engine interface {
Table(tableNameOrBean interface{}) *xorm.Session
Count(...interface{}) (int64, error)
Decr(column string, arg ...interface{}) *xorm.Session
Delete(interface{}) (int64, error)
Delete(...interface{}) (int64, error)
Exec(...interface{}) (sql.Result, error)
Find(interface{}, ...interface{}) error
Get(interface{}) (bool, error)

View File

@@ -8,9 +8,12 @@ import (
"io/ioutil"
"os"
"path/filepath"
"strings"
"testing"
"code.gitea.io/gitea/modules/auth/oauth2"
"code.gitea.io/gitea/modules/setting"
"xorm.io/xorm/schemas"
"github.com/stretchr/testify/assert"
)
@@ -32,3 +35,26 @@ func TestDumpDatabase(t *testing.T) {
assert.NoError(t, DumpDatabase(filepath.Join(dir, dbType+".sql"), dbType))
}
}
func TestDumpLoginSource(t *testing.T) {
assert.NoError(t, PrepareTestDatabase())
loginSourceSchema, err := x.TableInfo(new(LoginSource))
assert.NoError(t, err)
CreateLoginSource(&LoginSource{
Type: LoginOAuth2,
Name: "TestSource",
IsActived: false,
Cfg: &OAuth2Config{
Provider: "TestSourceProvider",
CustomURLMapping: &oauth2.CustomURLMapping{},
},
})
sb := new(strings.Builder)
x.DumpTables([]*schemas.Table{loginSourceSchema}, sb)
assert.Contains(t, sb.String(), `"Provider":"TestSourceProvider"`)
}

View File

@@ -17,7 +17,7 @@ import (
"code.gitea.io/gitea/modules/timeutil"
"code.gitea.io/gitea/modules/util"
"github.com/dgrijalva/jwt-go"
"github.com/golang-jwt/jwt"
uuid "github.com/google/uuid"
"golang.org/x/crypto/bcrypt"
"xorm.io/xorm"

View File

@@ -28,7 +28,7 @@ type UnitConfig struct{}
// FromDB fills up a UnitConfig from serialized format.
func (cfg *UnitConfig) FromDB(bs []byte) error {
return jsonUnmarshalIgnoreErroneousBOM(bs, &cfg)
return jsonUnmarshalHandleDoubleEncode(bs, &cfg)
}
// ToDB exports a UnitConfig to a serialized format.
@@ -44,7 +44,7 @@ type ExternalWikiConfig struct {
// FromDB fills up a ExternalWikiConfig from serialized format.
func (cfg *ExternalWikiConfig) FromDB(bs []byte) error {
return jsonUnmarshalIgnoreErroneousBOM(bs, &cfg)
return jsonUnmarshalHandleDoubleEncode(bs, &cfg)
}
// ToDB exports a ExternalWikiConfig to a serialized format.
@@ -62,7 +62,7 @@ type ExternalTrackerConfig struct {
// FromDB fills up a ExternalTrackerConfig from serialized format.
func (cfg *ExternalTrackerConfig) FromDB(bs []byte) error {
return jsonUnmarshalIgnoreErroneousBOM(bs, &cfg)
return jsonUnmarshalHandleDoubleEncode(bs, &cfg)
}
// ToDB exports a ExternalTrackerConfig to a serialized format.
@@ -80,7 +80,7 @@ type IssuesConfig struct {
// FromDB fills up a IssuesConfig from serialized format.
func (cfg *IssuesConfig) FromDB(bs []byte) error {
return jsonUnmarshalIgnoreErroneousBOM(bs, &cfg)
return jsonUnmarshalHandleDoubleEncode(bs, &cfg)
}
// ToDB exports a IssuesConfig to a serialized format.
@@ -102,7 +102,7 @@ type PullRequestsConfig struct {
// FromDB fills up a PullRequestsConfig from serialized format.
func (cfg *PullRequestsConfig) FromDB(bs []byte) error {
return jsonUnmarshalIgnoreErroneousBOM(bs, &cfg)
return jsonUnmarshalHandleDoubleEncode(bs, &cfg)
}
// ToDB exports a PullRequestsConfig to a serialized format.

View File

@@ -12,7 +12,7 @@ import (
"math/big"
"time"
"github.com/dgrijalva/jwt-go"
"github.com/golang-jwt/jwt"
)
// GetRandomString generate random string by specify chars.

View File

@@ -7,6 +7,7 @@ package git
import (
"bufio"
"bytes"
"context"
"io"
"math"
"strconv"
@@ -15,20 +16,24 @@ import (
// CatFileBatch opens git cat-file --batch in the provided repo and returns a stdin pipe, a stdout reader and cancel function
func CatFileBatch(repoPath string) (*io.PipeWriter, *bufio.Reader, func()) {
// Next feed the commits in order into cat-file --batch, followed by their trees and sub trees as necessary.
// We often want to feed the commits in order into cat-file --batch, followed by their trees and sub trees as necessary.
// so let's create a batch stdin and stdout
batchStdinReader, batchStdinWriter := io.Pipe()
batchStdoutReader, batchStdoutWriter := io.Pipe()
ctx, ctxCancel := context.WithCancel(DefaultContext)
closed := make(chan struct{})
cancel := func() {
_ = batchStdinReader.Close()
_ = batchStdinWriter.Close()
_ = batchStdoutReader.Close()
_ = batchStdoutWriter.Close()
ctxCancel()
<-closed
}
go func() {
stderr := strings.Builder{}
err := NewCommand("cat-file", "--batch").RunInDirFullPipeline(repoPath, batchStdoutWriter, &stderr, batchStdinReader)
err := NewCommandContext(ctx, "cat-file", "--batch").RunInDirFullPipeline(repoPath, batchStdoutWriter, &stderr, batchStdinReader)
if err != nil {
_ = batchStdoutWriter.CloseWithError(ConcatenateError(err, (&stderr).String()))
_ = batchStdinReader.CloseWithError(ConcatenateError(err, (&stderr).String()))
@@ -36,10 +41,11 @@ func CatFileBatch(repoPath string) (*io.PipeWriter, *bufio.Reader, func()) {
_ = batchStdoutWriter.Close()
_ = batchStdinReader.Close()
}
close(closed)
}()
// For simplicities sake we'll us a buffered reader to read from the cat-file --batch
batchReader := bufio.NewReader(batchStdoutReader)
batchReader := bufio.NewReaderSize(batchStdoutReader, 32*1024)
return batchStdinWriter, batchReader, cancel
}

View File

@@ -21,7 +21,7 @@ import (
"code.gitea.io/gitea/modules/setting"
"code.gitea.io/gitea/modules/storage"
"github.com/dgrijalva/jwt-go"
"github.com/golang-jwt/jwt"
jsoniter "github.com/json-iterator/go"
)

View File

@@ -92,7 +92,6 @@ func runMigrateTask(t *models.Task) (err error) {
}
opts.MigrateToRepoID = t.RepoID
var repo *models.Repository
ctx, cancel := context.WithCancel(graceful.GetManager().ShutdownContext())
defer cancel()
@@ -106,9 +105,9 @@ func runMigrateTask(t *models.Task) (err error) {
return
}
repo, err = migrations.MigrateRepository(ctx, t.Doer, t.Owner.Name, *opts)
t.Repo, err = migrations.MigrateRepository(ctx, t.Doer, t.Owner.Name, *opts)
if err == nil {
log.Trace("Repository migrated [%d]: %s/%s", repo.ID, t.Owner.Name, repo.Name)
log.Trace("Repository migrated [%d]: %s/%s", t.Repo.ID, t.Owner.Name, t.Repo.Name)
return
}

View File

@@ -6,10 +6,13 @@ package util
import (
"os"
"runtime"
"syscall"
"time"
)
const windowsSharingViolationError syscall.Errno = 32
// Remove removes the named file or (empty) directory with at most 5 attempts.
func Remove(name string) error {
var err error
@@ -25,6 +28,12 @@ func Remove(name string) error {
continue
}
if unwrapped == windowsSharingViolationError && runtime.GOOS == "windows" {
// try again
<-time.After(100 * time.Millisecond)
continue
}
if unwrapped == syscall.ENOENT {
// it's already gone
return nil
@@ -48,6 +57,12 @@ func RemoveAll(name string) error {
continue
}
if unwrapped == windowsSharingViolationError && runtime.GOOS == "windows" {
// try again
<-time.After(100 * time.Millisecond)
continue
}
if unwrapped == syscall.ENOENT {
// it's already gone
return nil
@@ -64,13 +79,19 @@ func Rename(oldpath, newpath string) error {
if err == nil {
break
}
unwrapped := err.(*os.PathError).Err
unwrapped := err.(*os.LinkError).Err
if unwrapped == syscall.EBUSY || unwrapped == syscall.ENOTEMPTY || unwrapped == syscall.EPERM || unwrapped == syscall.EMFILE || unwrapped == syscall.ENFILE {
// try again
<-time.After(100 * time.Millisecond)
continue
}
if unwrapped == windowsSharingViolationError && runtime.GOOS == "windows" {
// try again
<-time.After(100 * time.Millisecond)
continue
}
if i == 0 && os.IsNotExist(err) {
return err
}

View File

@@ -278,6 +278,7 @@ func Diff(ctx *context.Context) {
ctx.ServerError("Repo.GitRepo.GetCommit", err)
return
}
defer gitRepo.Close()
repoPath = ctx.Repo.Repository.WikiPath()
} else {
gitRepo = ctx.Repo.GitRepo

View File

@@ -131,6 +131,9 @@ func wikiContentsByName(ctx *context.Context, commit *git.Commit, wikiName strin
func renderViewPage(ctx *context.Context) (*git.Repository, *git.TreeEntry) {
wikiRepo, commit, err := findWikiRepoCommit(ctx)
if err != nil {
if wikiRepo != nil {
wikiRepo.Close()
}
if !git.IsErrNotExist(err) {
ctx.ServerError("GetBranchCommit", err)
}
@@ -354,17 +357,14 @@ func Wiki(ctx *context.Context) {
}
wikiRepo, entry := renderViewPage(ctx)
if ctx.Written() {
if wikiRepo != nil {
wikiRepo.Close()
}
return
}
defer func() {
if wikiRepo != nil {
wikiRepo.Close()
}
}()
if ctx.Written() {
return
}
if entry == nil {
ctx.Data["Title"] = ctx.Tr("repo.wiki")
ctx.HTML(200, tplWikiStart)
@@ -399,17 +399,15 @@ func WikiRevision(ctx *context.Context) {
}
wikiRepo, entry := renderRevisionPage(ctx)
if ctx.Written() {
if wikiRepo != nil {
wikiRepo.Close()
}
return
}
defer func() {
if wikiRepo != nil {
wikiRepo.Close()
}
}()
if ctx.Written() {
return
}
if entry == nil {
ctx.Data["Title"] = ctx.Tr("repo.wiki")
ctx.HTML(200, tplWikiStart)
@@ -446,13 +444,14 @@ func WikiPages(ctx *context.Context) {
}
return
}
entries, err := commit.ListEntries()
if err != nil {
defer func() {
if wikiRepo != nil {
wikiRepo.Close()
}
}()
entries, err := commit.ListEntries()
if err != nil {
ctx.ServerError("ListEntries", err)
return
}
@@ -463,10 +462,6 @@ func WikiPages(ctx *context.Context) {
}
c, err := wikiRepo.GetCommitByPath(entry.Name())
if err != nil {
if wikiRepo != nil {
wikiRepo.Close()
}
ctx.ServerError("GetCommit", err)
return
}
@@ -475,10 +470,6 @@ func WikiPages(ctx *context.Context) {
if models.IsErrWikiInvalidFileName(err) {
continue
}
if wikiRepo != nil {
wikiRepo.Close()
}
ctx.ServerError("WikiFilenameToName", err)
return
}
@@ -490,21 +481,25 @@ func WikiPages(ctx *context.Context) {
}
ctx.Data["Pages"] = pages
defer func() {
if wikiRepo != nil {
wikiRepo.Close()
}
}()
ctx.HTML(200, tplWikiPages)
}
// WikiRaw outputs raw blob requested by user (image for example)
func WikiRaw(ctx *context.Context) {
wikiRepo, commit, err := findWikiRepoCommit(ctx)
if err != nil {
defer func() {
if wikiRepo != nil {
wikiRepo.Close()
}
}()
if err != nil {
if git.IsErrNotExist(err) {
ctx.NotFound("findEntryForFile", nil)
return
}
ctx.ServerError("findEntryForfile", err)
return
}
providedPath := ctx.Params("*")
@@ -520,9 +515,7 @@ func WikiRaw(ctx *context.Context) {
if entry == nil {
// Try to find a wiki page with that name
if strings.HasSuffix(providedPath, ".md") {
providedPath = providedPath[:len(providedPath)-3]
}
providedPath = strings.TrimSuffix(providedPath, ".md")
wikiPath := wiki_service.NameToFilename(providedPath)
entry, err = findEntryForFile(commit, wikiPath)

View File

@@ -912,7 +912,7 @@ func RegisterRoutes(m *web.Route) {
m.Get("/_pages", repo.WikiPages)
m.Get("/{page}/_revision", repo.WikiRevision)
m.Get("/commit/{sha:[a-f0-9]{7,40}}", repo.SetEditorconfigIfExists, repo.SetDiffViewStyle, repo.SetWhitespaceBehavior, repo.Diff)
m.Get("/commit/{sha:[a-f0-9]{7,40}}.{:patch|diff}", repo.RawDiff)
m.Get("/commit/{sha:[a-f0-9]{7,40}}.{ext:patch|diff}", repo.RawDiff)
m.Group("", func() {
m.Combo("/_new").Get(repo.NewWiki).

View File

@@ -21,7 +21,7 @@ import (
"code.gitea.io/gitea/modules/web"
"gitea.com/go-chi/binding"
"github.com/dgrijalva/jwt-go"
"github.com/golang-jwt/jwt"
)
const (

8
vendor/github.com/denisenkom/go-mssqldb/.gitignore generated vendored Normal file
View File

@@ -0,0 +1,8 @@
/.idea
/.connstr
.vscode
.terraform
*.tfstate*
*.log
*.swp
*~

10
vendor/github.com/denisenkom/go-mssqldb/.golangci.yml generated vendored Normal file
View File

@@ -0,0 +1,10 @@
linters:
enable:
# basic go linters
- gofmt
- golint
- govet
# sql related linters
- rowserrcheck
- sqlclosecheck

View File

@@ -6,19 +6,8 @@ import (
"context"
"database/sql/driver"
"errors"
"fmt"
)
var _ driver.Connector = &accessTokenConnector{}
// accessTokenConnector wraps Connector and injects a
// fresh access token when connecting to the database
type accessTokenConnector struct {
Connector
accessTokenProvider func() (string, error)
}
// NewAccessTokenConnector creates a new connector from a DSN and a token provider.
// The token provider func will be called when a new connection is requested and should return a valid access token.
// The returned connector may be used with sql.OpenDB.
@@ -32,20 +21,10 @@ func NewAccessTokenConnector(dsn string, tokenProvider func() (string, error)) (
return nil, err
}
c := &accessTokenConnector{
Connector: *conn,
accessTokenProvider: tokenProvider,
}
return c, nil
}
// Connect returns a new database connection
func (c *accessTokenConnector) Connect(ctx context.Context) (driver.Conn, error) {
var err error
c.Connector.params.fedAuthAccessToken, err = c.accessTokenProvider()
if err != nil {
return nil, fmt.Errorf("mssql: error retrieving access token: %+v", err)
conn.params.fedAuthLibrary = fedAuthLibrarySecurityToken
conn.securityTokenProvider = func(ctx context.Context) (string, error) {
return tokenProvider()
}
return c.Connector.Connect(ctx)
return conn, nil
}

View File

@@ -39,6 +39,9 @@ environment:
- APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2019
GOVERSION: 115
SQLINSTANCE: SQL2017
- APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2019
GOVERSION: 116
SQLINSTANCE: SQL2017
install:
- set GOROOT=c:\go%GOVERSION%

View File

@@ -48,8 +48,8 @@ type tdsBuffer struct {
func newTdsBuffer(bufsize uint16, transport io.ReadWriteCloser) *tdsBuffer {
return &tdsBuffer{
packetSize: int(bufsize),
wbuf: make([]byte, 1<<16),
rbuf: make([]byte, 1<<16),
wbuf: make([]byte, bufsize),
rbuf: make([]byte, bufsize),
rpos: 8,
transport: transport,
}
@@ -137,19 +137,28 @@ func (w *tdsBuffer) FinishPacket() error {
var headerSize = binary.Size(header{})
func (r *tdsBuffer) readNextPacket() error {
h := header{}
var err error
err = binary.Read(r.transport, binary.BigEndian, &h)
buf := r.rbuf[:headerSize]
_, err := io.ReadFull(r.transport, buf)
if err != nil {
return err
}
h := header{
PacketType: packetType(buf[0]),
Status: buf[1],
Size: binary.BigEndian.Uint16(buf[2:4]),
Spid: binary.BigEndian.Uint16(buf[4:6]),
PacketNo: buf[6],
Pad: buf[7],
}
if int(h.Size) > r.packetSize {
return errors.New("Invalid packet size, it is longer than buffer size")
return errors.New("invalid packet size, it is longer than buffer size")
}
if headerSize > int(h.Size) {
return errors.New("Invalid packet size, it is shorter than header size")
return errors.New("invalid packet size, it is shorter than header size")
}
_, err = io.ReadFull(r.transport, r.rbuf[headerSize:h.Size])
//s := base64.StdEncoding.EncodeToString(r.rbuf[headerSize:h.Size])
//fmt.Print(s)
if err != nil {
return err
}

View File

@@ -44,8 +44,9 @@ type BulkOptions struct {
type DataValue interface{}
const (
sqlDateFormat = "2006-01-02"
sqlTimeFormat = "2006-01-02 15:04:05.999999999Z07:00"
sqlDateFormat = "2006-01-02"
sqlDateTimeFormat = "2006-01-02 15:04:05.999999999Z07:00"
sqlTimeFormat = "15:04:05.9999999"
)
func (cn *Conn) CreateBulk(table string, columns []string) (_ *Bulk) {
@@ -86,7 +87,7 @@ func (b *Bulk) sendBulkCommand(ctx context.Context) (err error) {
b.bulkColumns = append(b.bulkColumns, *bulkCol)
b.dlogf("Adding column %s %s %#x", colname, bulkCol.ColName, bulkCol.ti.TypeId)
} else {
return fmt.Errorf("Column %s does not exist in destination table %s", colname, b.tablename)
return fmt.Errorf("column %s does not exist in destination table %s", colname, b.tablename)
}
}
@@ -166,7 +167,7 @@ func (b *Bulk) AddRow(row []interface{}) (err error) {
}
if len(row) != len(b.bulkColumns) {
return fmt.Errorf("Row does not have the same number of columns than the destination table %d %d",
return fmt.Errorf("row does not have the same number of columns than the destination table %d %d",
len(row), len(b.bulkColumns))
}
@@ -215,7 +216,7 @@ func (b *Bulk) makeRowData(row []interface{}) ([]byte, error) {
}
func (b *Bulk) Done() (rowcount int64, err error) {
if b.headerSent == false {
if !b.headerSent {
//no rows had been sent
return 0, nil
}
@@ -233,24 +234,13 @@ func (b *Bulk) Done() (rowcount int64, err error) {
buf.FinishPacket()
tokchan := make(chan tokenStruct, 5)
go processResponse(b.ctx, b.cn.sess, tokchan, nil)
var rowCount int64
for token := range tokchan {
switch token := token.(type) {
case doneStruct:
if token.Status&doneCount != 0 {
rowCount = int64(token.RowCount)
}
if token.isError() {
return 0, token.getError()
}
case error:
return 0, b.cn.checkBadConn(token)
}
reader := startReading(b.cn.sess, b.ctx, nil)
err = reader.iterateResponse()
if err != nil {
return 0, b.cn.checkBadConn(err)
}
return rowCount, nil
return reader.rowCount, nil
}
func (b *Bulk) createColMetadata() []byte {
@@ -421,7 +411,7 @@ func (b *Bulk) makeParam(val DataValue, col columnStruct) (res param, err error)
res.ti.Size = len(res.buffer)
case string:
var t time.Time
if t, err = time.Parse(sqlTimeFormat, val); err != nil {
if t, err = time.Parse(sqlDateTimeFormat, val); err != nil {
return res, fmt.Errorf("bulk: unable to convert string to date: %v", err)
}
res.buffer = encodeDateTime2(t, int(col.ti.Scale))
@@ -437,7 +427,7 @@ func (b *Bulk) makeParam(val DataValue, col columnStruct) (res param, err error)
res.ti.Size = len(res.buffer)
case string:
var t time.Time
if t, err = time.Parse(sqlTimeFormat, val); err != nil {
if t, err = time.Parse(sqlDateTimeFormat, val); err != nil {
return res, fmt.Errorf("bulk: unable to convert string to date: %v", err)
}
res.buffer = encodeDateTimeOffset(t, int(col.ti.Scale))
@@ -468,7 +458,7 @@ func (b *Bulk) makeParam(val DataValue, col columnStruct) (res param, err error)
case time.Time:
t = val
case string:
if t, err = time.Parse(sqlTimeFormat, val); err != nil {
if t, err = time.Parse(sqlDateTimeFormat, val); err != nil {
return res, fmt.Errorf("bulk: unable to convert string to date: %v", err)
}
default:
@@ -485,7 +475,22 @@ func (b *Bulk) makeParam(val DataValue, col columnStruct) (res param, err error)
} else {
err = fmt.Errorf("mssql: invalid size of column %d", col.ti.Size)
}
case typeTimeN:
var t time.Time
switch val := val.(type) {
case time.Time:
res.buffer = encodeTime(val.Hour(), val.Minute(), val.Second(), val.Nanosecond(), int(col.ti.Scale))
res.ti.Size = len(res.buffer)
case string:
if t, err = time.Parse(sqlTimeFormat, val); err != nil {
return res, fmt.Errorf("bulk: unable to convert string to time: %v", err)
}
res.buffer = encodeTime(t.Hour(), t.Minute(), t.Second(), t.Nanosecond(), int(col.ti.Scale))
res.ti.Size = len(res.buffer)
default:
err = fmt.Errorf("mssql: invalid type for time column: %T %s", val, val)
return
}
// case typeMoney, typeMoney4, typeMoneyN:
case typeDecimal, typeDecimalN, typeNumeric, typeNumericN:
prec := col.ti.Prec

View File

@@ -37,11 +37,17 @@ type connectParams struct {
failOverPartner string
failOverPort uint64
packetSize uint16
fedAuthAccessToken string
fedAuthLibrary int
fedAuthADALWorkflow byte
}
// default packet size for TDS buffer
const defaultPacketSize = 4096
func parseConnectParams(dsn string) (connectParams, error) {
var p connectParams
p := connectParams{
fedAuthLibrary: fedAuthLibraryReserved,
}
var params map[string]string
if strings.HasPrefix(dsn, "odbc:") {
@@ -65,7 +71,7 @@ func parseConnectParams(dsn string) (connectParams, error) {
var err error
p.logFlags, err = strconv.ParseUint(strlog, 10, 64)
if err != nil {
return p, fmt.Errorf("Invalid log parameter '%s': %s", strlog, err.Error())
return p, fmt.Errorf("invalid log parameter '%s': %s", strlog, err.Error())
}
}
server := params["server"]
@@ -87,20 +93,19 @@ func parseConnectParams(dsn string) (connectParams, error) {
var err error
p.port, err = strconv.ParseUint(strport, 10, 16)
if err != nil {
f := "Invalid tcp port '%v': %v"
f := "invalid tcp port '%v': %v"
return p, fmt.Errorf(f, strport, err.Error())
}
}
// https://docs.microsoft.com/en-us/sql/database-engine/configure-windows/configure-the-network-packet-size-server-configuration-option
// Default packet size remains at 4096 bytes
p.packetSize = 4096
p.packetSize = defaultPacketSize
strpsize, ok := params["packet size"]
if ok {
var err error
psize, err := strconv.ParseUint(strpsize, 0, 16)
if err != nil {
f := "Invalid packet size '%v': %v"
f := "invalid packet size '%v': %v"
return p, fmt.Errorf(f, strpsize, err.Error())
}
@@ -123,7 +128,7 @@ func parseConnectParams(dsn string) (connectParams, error) {
if strconntimeout, ok := params["connection timeout"]; ok {
timeout, err := strconv.ParseUint(strconntimeout, 10, 64)
if err != nil {
f := "Invalid connection timeout '%v': %v"
f := "invalid connection timeout '%v': %v"
return p, fmt.Errorf(f, strconntimeout, err.Error())
}
p.conn_timeout = time.Duration(timeout) * time.Second
@@ -132,7 +137,7 @@ func parseConnectParams(dsn string) (connectParams, error) {
if strdialtimeout, ok := params["dial timeout"]; ok {
timeout, err := strconv.ParseUint(strdialtimeout, 10, 64)
if err != nil {
f := "Invalid dial timeout '%v': %v"
f := "invalid dial timeout '%v': %v"
return p, fmt.Errorf(f, strdialtimeout, err.Error())
}
p.dial_timeout = time.Duration(timeout) * time.Second
@@ -144,7 +149,7 @@ func parseConnectParams(dsn string) (connectParams, error) {
if keepAlive, ok := params["keepalive"]; ok {
timeout, err := strconv.ParseUint(keepAlive, 10, 64)
if err != nil {
f := "Invalid keepAlive value '%s': %s"
f := "invalid keepAlive value '%s': %s"
return p, fmt.Errorf(f, keepAlive, err.Error())
}
p.keepAlive = time.Duration(timeout) * time.Second
@@ -157,7 +162,7 @@ func parseConnectParams(dsn string) (connectParams, error) {
var err error
p.encrypt, err = strconv.ParseBool(encrypt)
if err != nil {
f := "Invalid encrypt '%s': %s"
f := "invalid encrypt '%s': %s"
return p, fmt.Errorf(f, encrypt, err.Error())
}
}
@@ -169,7 +174,7 @@ func parseConnectParams(dsn string) (connectParams, error) {
var err error
p.trustServerCertificate, err = strconv.ParseBool(trust)
if err != nil {
f := "Invalid trust server certificate '%s': %s"
f := "invalid trust server certificate '%s': %s"
return p, fmt.Errorf(f, trust, err.Error())
}
}
@@ -209,7 +214,7 @@ func parseConnectParams(dsn string) (connectParams, error) {
if ok {
if appintent == "ReadOnly" {
if p.database == "" {
return p, fmt.Errorf("Database must be specified when ApplicationIntent is ReadOnly")
return p, fmt.Errorf("database must be specified when ApplicationIntent is ReadOnly")
}
p.typeFlags |= fReadOnlyIntent
}
@@ -225,7 +230,7 @@ func parseConnectParams(dsn string) (connectParams, error) {
var err error
p.failOverPort, err = strconv.ParseUint(failOverPort, 0, 16)
if err != nil {
f := "Invalid tcp port '%v': %v"
f := "invalid tcp port '%v': %v"
return p, fmt.Errorf(f, failOverPort, err.Error())
}
}
@@ -233,6 +238,30 @@ func parseConnectParams(dsn string) (connectParams, error) {
return p, nil
}
// convert connectionParams to url style connection string
// used mostly for testing
func (p connectParams) toUrl() *url.URL {
q := url.Values{}
if p.database != "" {
q.Add("database", p.database)
}
if p.logFlags != 0 {
q.Add("log", strconv.FormatUint(p.logFlags, 10))
}
res := url.URL{
Scheme: "sqlserver",
Host: p.host,
User: url.UserPassword(p.user, p.password),
}
if p.instance != "" {
res.Path = p.instance
}
if len(q) > 0 {
res.RawQuery = q.Encode()
}
return &res
}
func splitConnectionString(dsn string) (res map[string]string) {
res = map[string]string{}
parts := strings.Split(dsn, ";")
@@ -340,7 +369,7 @@ func splitConnectionStringOdbc(dsn string) (map[string]string, error) {
case parserStateBeforeKey:
switch {
case c == '=':
return res, fmt.Errorf("Unexpected character = at index %d. Expected start of key or semi-colon or whitespace.", i)
return res, fmt.Errorf("unexpected character = at index %d. Expected start of key or semi-colon or whitespace", i)
case !unicode.IsSpace(c) && c != ';':
state = parserStateKey
key += string(c)
@@ -419,7 +448,7 @@ func splitConnectionStringOdbc(dsn string) (map[string]string, error) {
case unicode.IsSpace(c):
// Ignore whitespace
default:
return res, fmt.Errorf("Unexpected character %c at index %d. Expected semi-colon or whitespace.", c, i)
return res, fmt.Errorf("unexpected character %c at index %d. Expected semi-colon or whitespace", c, i)
}
case parserStateEndValue:
@@ -429,7 +458,7 @@ func splitConnectionStringOdbc(dsn string) (map[string]string, error) {
case unicode.IsSpace(c):
// Ignore whitespace
default:
return res, fmt.Errorf("Unexpected character %c at index %d. Expected semi-colon or whitespace.", c, i)
return res, fmt.Errorf("unexpected character %c at index %d. Expected semi-colon or whitespace", c, i)
}
}
}
@@ -444,7 +473,7 @@ func splitConnectionStringOdbc(dsn string) (map[string]string, error) {
case parserStateBareValue:
res[key] = strings.TrimRightFunc(value, unicode.IsSpace)
case parserStateBracedValue:
return res, fmt.Errorf("Unexpected end of braced value at index %d.", len(dsn))
return res, fmt.Errorf("unexpected end of braced value at index %d", len(dsn))
case parserStateBracedValueClosingBrace: // End of braced value
res[key] = value
case parserStateEndValue: // Okay

82
vendor/github.com/denisenkom/go-mssqldb/fedauth.go generated vendored Normal file
View File

@@ -0,0 +1,82 @@
package mssql
import (
"context"
"errors"
)
// Federated authentication library affects the login data structure and message sequence.
const (
// fedAuthLibraryLiveIDCompactToken specifies the Microsoft Live ID Compact Token authentication scheme
fedAuthLibraryLiveIDCompactToken = 0x00
// fedAuthLibrarySecurityToken specifies a token-based authentication where the token is available
// without additional information provided during the login sequence.
fedAuthLibrarySecurityToken = 0x01
// fedAuthLibraryADAL specifies a token-based authentication where a token is obtained during the
// login sequence using the server SPN and STS URL provided by the server during login.
fedAuthLibraryADAL = 0x02
// fedAuthLibraryReserved is used to indicate that no federated authentication scheme applies.
fedAuthLibraryReserved = 0x7F
)
// Federated authentication ADAL workflow affects the mechanism used to authenticate.
const (
// fedAuthADALWorkflowPassword uses a username/password to obtain a token from Active Directory
fedAuthADALWorkflowPassword = 0x01
// fedAuthADALWorkflowPassword uses the Windows identity to obtain a token from Active Directory
fedAuthADALWorkflowIntegrated = 0x02
// fedAuthADALWorkflowMSI uses the managed identity service to obtain a token
fedAuthADALWorkflowMSI = 0x03
)
// newSecurityTokenConnector creates a new connector from a DSN and a token provider.
// When invoked, token provider implementations should contact the security token
// service specified and obtain the appropriate token, or return an error
// to indicate why a token is not available.
// The returned connector may be used with sql.OpenDB.
func newSecurityTokenConnector(dsn string, tokenProvider func(ctx context.Context) (string, error)) (*Connector, error) {
if tokenProvider == nil {
return nil, errors.New("mssql: tokenProvider cannot be nil")
}
conn, err := NewConnector(dsn)
if err != nil {
return nil, err
}
conn.params.fedAuthLibrary = fedAuthLibrarySecurityToken
conn.securityTokenProvider = tokenProvider
return conn, nil
}
// newADALTokenConnector creates a new connector from a DSN and a Active Directory token provider.
// Token provider implementations are called during federated
// authentication login sequences where the server provides a service
// principal name and security token service endpoint that should be used
// to obtain the token. Implementations should contact the security token
// service specified and obtain the appropriate token, or return an error
// to indicate why a token is not available.
//
// The returned connector may be used with sql.OpenDB.
func newActiveDirectoryTokenConnector(dsn string, adalWorkflow byte, tokenProvider func(ctx context.Context, serverSPN, stsURL string) (string, error)) (*Connector, error) {
if tokenProvider == nil {
return nil, errors.New("mssql: tokenProvider cannot be nil")
}
conn, err := NewConnector(dsn)
if err != nil {
return nil, err
}
conn.params.fedAuthLibrary = fedAuthLibraryADAL
conn.params.fedAuthADALWorkflow = adalWorkflow
conn.adalTokenProvider = tokenProvider
return conn, nil
}

View File

@@ -58,6 +58,7 @@ func (d *Driver) OpenConnector(dsn string) (*Connector, error) {
if err != nil {
return nil, err
}
return &Connector{
params: params,
driver: d,
@@ -100,6 +101,12 @@ type Connector struct {
params connectParams
driver *Driver
// callback that can provide a security token during login
securityTokenProvider func(ctx context.Context) (string, error)
// callback that can provide a security token during ADAL login
adalTokenProvider func(ctx context.Context, serverSPN, stsURL string) (string, error)
// SessionInitSQL is executed after marking a given session to be reset.
// When not present, the next query will still reset the session to the
// database defaults.
@@ -148,15 +155,7 @@ type Conn struct {
processQueryText bool
connectionGood bool
outs map[string]interface{}
returnStatus *ReturnStatus
}
func (c *Conn) setReturnStatus(s ReturnStatus) {
if c.returnStatus == nil {
return
}
*c.returnStatus = s
outs map[string]interface{}
}
func (c *Conn) checkBadConn(err error) error {
@@ -201,20 +200,15 @@ func (c *Conn) clearOuts() {
}
func (c *Conn) simpleProcessResp(ctx context.Context) error {
tokchan := make(chan tokenStruct, 5)
go processResponse(ctx, c.sess, tokchan, c.outs)
reader := startReading(c.sess, ctx, c.outs)
c.clearOuts()
for tok := range tokchan {
switch token := tok.(type) {
case doneStruct:
if token.isError() {
return c.checkBadConn(token.getError())
}
case error:
return c.checkBadConn(token)
}
var resultError error
err := reader.iterateResponse()
if err != nil {
return c.checkBadConn(err)
}
return nil
return resultError
}
func (c *Conn) Commit() error {
@@ -239,7 +233,7 @@ func (c *Conn) sendCommitRequest() error {
c.sess.log.Printf("Failed to send CommitXact with %v", err)
}
c.connectionGood = false
return fmt.Errorf("Faild to send CommitXact: %v", err)
return fmt.Errorf("faild to send CommitXact: %v", err)
}
return nil
}
@@ -266,7 +260,7 @@ func (c *Conn) sendRollbackRequest() error {
c.sess.log.Printf("Failed to send RollbackXact with %v", err)
}
c.connectionGood = false
return fmt.Errorf("Failed to send RollbackXact: %v", err)
return fmt.Errorf("failed to send RollbackXact: %v", err)
}
return nil
}
@@ -303,7 +297,7 @@ func (c *Conn) sendBeginRequest(ctx context.Context, tdsIsolation isoLevel) erro
c.sess.log.Printf("Failed to send BeginXact with %v", err)
}
c.connectionGood = false
return fmt.Errorf("Failed to send BeginXact: %v", err)
return fmt.Errorf("failed to send BeginXact: %v", err)
}
return nil
}
@@ -478,7 +472,7 @@ func (s *Stmt) sendQuery(args []namedValue) (err error) {
conn.sess.log.Printf("Failed to send Rpc with %v", err)
}
conn.connectionGood = false
return fmt.Errorf("Failed to send RPC: %v", err)
return fmt.Errorf("failed to send RPC: %v", err)
}
}
return
@@ -595,38 +589,46 @@ func (s *Stmt) queryContext(ctx context.Context, args []namedValue) (rows driver
}
func (s *Stmt) processQueryResponse(ctx context.Context) (res driver.Rows, err error) {
tokchan := make(chan tokenStruct, 5)
ctx, cancel := context.WithCancel(ctx)
go processResponse(ctx, s.c.sess, tokchan, s.c.outs)
reader := startReading(s.c.sess, ctx, s.c.outs)
s.c.clearOuts()
// process metadata
var cols []columnStruct
loop:
for tok := range tokchan {
switch token := tok.(type) {
// By ignoring DONE token we effectively
// skip empty result-sets.
// This improves results in queries like that:
// set nocount on; select 1
// see TestIgnoreEmptyResults test
//case doneStruct:
//break loop
case []columnStruct:
cols = token
break loop
case doneStruct:
if token.isError() {
cancel()
return nil, s.c.checkBadConn(token.getError())
for {
tok, err := reader.nextToken()
if err == nil {
if tok == nil {
break
} else {
switch token := tok.(type) {
// By ignoring DONE token we effectively
// skip empty result-sets.
// This improves results in queries like that:
// set nocount on; select 1
// see TestIgnoreEmptyResults test
//case doneStruct:
//break loop
case []columnStruct:
cols = token
break loop
case doneStruct:
if token.isError() {
// need to cleanup cancellable context
cancel()
return nil, s.c.checkBadConn(token.getError())
}
case ReturnStatus:
s.c.sess.setReturnStatus(token)
}
}
case ReturnStatus:
s.c.setReturnStatus(token)
case error:
} else {
// need to cleanup cancellable context
cancel()
return nil, s.c.checkBadConn(token)
return nil, s.c.checkBadConn(err)
}
}
res = &Rows{stmt: s, tokchan: tokchan, cols: cols, cancel: cancel}
res = &Rows{stmt: s, reader: reader, cols: cols, cancel: cancel}
return
}
@@ -648,48 +650,46 @@ func (s *Stmt) exec(ctx context.Context, args []namedValue) (res driver.Result,
}
func (s *Stmt) processExec(ctx context.Context) (res driver.Result, err error) {
tokchan := make(chan tokenStruct, 5)
go processResponse(ctx, s.c.sess, tokchan, s.c.outs)
reader := startReading(s.c.sess, ctx, s.c.outs)
s.c.clearOuts()
var rowCount int64
for token := range tokchan {
switch token := token.(type) {
case doneInProcStruct:
if token.Status&doneCount != 0 {
rowCount += int64(token.RowCount)
}
case doneStruct:
if token.Status&doneCount != 0 {
rowCount += int64(token.RowCount)
}
if token.isError() {
return nil, token.getError()
}
case ReturnStatus:
s.c.setReturnStatus(token)
case error:
return nil, token
}
err = reader.iterateResponse()
if err != nil {
return nil, s.c.checkBadConn(err)
}
return &Result{s.c, rowCount}, nil
return &Result{s.c, reader.rowCount}, nil
}
type Rows struct {
stmt *Stmt
cols []columnStruct
tokchan chan tokenStruct
stmt *Stmt
cols []columnStruct
reader *tokenProcessor
nextCols []columnStruct
cancel func()
}
func (rc *Rows) Close() error {
// need to add a test which returns lots of rows
// and check closing after reading only few rows
rc.cancel()
for _ = range rc.tokchan {
for {
tok, err := rc.reader.nextToken()
if err == nil {
if tok == nil {
return nil
} else {
// continue consuming tokens
continue
}
} else {
if err == rc.reader.ctx.Err() {
return nil
} else {
return err
}
}
}
rc.tokchan = nil
return nil
}
func (rc *Rows) Columns() (res []string) {
@@ -707,27 +707,34 @@ func (rc *Rows) Next(dest []driver.Value) error {
if rc.nextCols != nil {
return io.EOF
}
for tok := range rc.tokchan {
switch tokdata := tok.(type) {
case []columnStruct:
rc.nextCols = tokdata
return io.EOF
case []interface{}:
for i := range dest {
dest[i] = tokdata[i]
for {
tok, err := rc.reader.nextToken()
if err == nil {
if tok == nil {
return io.EOF
} else {
switch tokdata := tok.(type) {
case []columnStruct:
rc.nextCols = tokdata
return io.EOF
case []interface{}:
for i := range dest {
dest[i] = tokdata[i]
}
return nil
case doneStruct:
if tokdata.isError() {
return rc.stmt.c.checkBadConn(tokdata.getError())
}
case ReturnStatus:
rc.stmt.c.sess.setReturnStatus(tokdata)
}
}
return nil
case doneStruct:
if tokdata.isError() {
return rc.stmt.c.checkBadConn(tokdata.getError())
}
case ReturnStatus:
rc.stmt.c.setReturnStatus(tokdata)
case error:
return rc.stmt.c.checkBadConn(tokdata)
} else {
return rc.stmt.c.checkBadConn(err)
}
}
return io.EOF
}
func (rc *Rows) HasNextResultSet() bool {
@@ -895,35 +902,41 @@ func (c *Conn) Ping(ctx context.Context) error {
var _ driver.ConnBeginTx = &Conn{}
func convertIsolationLevel(level sql.IsolationLevel) (isoLevel, error) {
switch level {
case sql.LevelDefault:
return isolationUseCurrent, nil
case sql.LevelReadUncommitted:
return isolationReadUncommited, nil
case sql.LevelReadCommitted:
return isolationReadCommited, nil
case sql.LevelWriteCommitted:
return isolationUseCurrent, errors.New("LevelWriteCommitted isolation level is not supported")
case sql.LevelRepeatableRead:
return isolationRepeatableRead, nil
case sql.LevelSnapshot:
return isolationSnapshot, nil
case sql.LevelSerializable:
return isolationSerializable, nil
case sql.LevelLinearizable:
return isolationUseCurrent, errors.New("LevelLinearizable isolation level is not supported")
default:
return isolationUseCurrent, errors.New("isolation level is not supported or unknown")
}
}
// BeginTx satisfies ConnBeginTx.
func (c *Conn) BeginTx(ctx context.Context, opts driver.TxOptions) (driver.Tx, error) {
if !c.connectionGood {
return nil, driver.ErrBadConn
}
if opts.ReadOnly {
return nil, errors.New("Read-only transactions are not supported")
return nil, errors.New("read-only transactions are not supported")
}
var tdsIsolation isoLevel
switch sql.IsolationLevel(opts.Isolation) {
case sql.LevelDefault:
tdsIsolation = isolationUseCurrent
case sql.LevelReadUncommitted:
tdsIsolation = isolationReadUncommited
case sql.LevelReadCommitted:
tdsIsolation = isolationReadCommited
case sql.LevelWriteCommitted:
return nil, errors.New("LevelWriteCommitted isolation level is not supported")
case sql.LevelRepeatableRead:
tdsIsolation = isolationRepeatableRead
case sql.LevelSnapshot:
tdsIsolation = isolationSnapshot
case sql.LevelSerializable:
tdsIsolation = isolationSerializable
case sql.LevelLinearizable:
return nil, errors.New("LevelLinearizable isolation level is not supported")
default:
return nil, errors.New("Isolation level is not supported or unknown")
tdsIsolation, err := convertIsolationLevel(sql.IsolationLevel(opts.Isolation))
if err != nil {
return nil, err
}
return c.begin(ctx, tdsIsolation)
}

View File

@@ -48,5 +48,5 @@ func (c *Connector) Driver() driver.Driver {
}
func (r *Result) LastInsertId() (int64, error) {
return -1, errors.New("LastInsertId is not supported. Please use the OUTPUT clause or add `select ID = convert(bigint, SCOPE_IDENTITY())` to the end of your query.")
return -1, errors.New("LastInsertId is not supported. Please use the OUTPUT clause or add `select ID = convert(bigint, SCOPE_IDENTITY())` to the end of your query")
}

View File

@@ -110,7 +110,7 @@ func (c *Conn) CheckNamedValue(nv *driver.NamedValue) error {
return nil
case *ReturnStatus:
*v = 0 // By default the return value should be zero.
c.returnStatus = v
c.sess.returnStatus = v
return driver.ErrRemoveArgument
case TVP:
return nil

View File

@@ -7,8 +7,8 @@ import (
)
type timeoutConn struct {
c net.Conn
timeout time.Duration
c net.Conn
timeout time.Duration
}
func newTimeoutConn(conn net.Conn, timeout time.Duration) *timeoutConn {
@@ -51,21 +51,21 @@ func (c timeoutConn) RemoteAddr() net.Addr {
}
func (c timeoutConn) SetDeadline(t time.Time) error {
panic("Not implemented")
return c.c.SetDeadline(t)
}
func (c timeoutConn) SetReadDeadline(t time.Time) error {
panic("Not implemented")
return c.c.SetReadDeadline(t)
}
func (c timeoutConn) SetWriteDeadline(t time.Time) error {
panic("Not implemented")
return c.c.SetWriteDeadline(t)
}
// this connection is used during TLS Handshake
// TDS protocol requires TLS handshake messages to be sent inside TDS packets
type tlsHandshakeConn struct {
buf *tdsBuffer
buf *tdsBuffer
packetPending bool
continueRead bool
}
@@ -75,7 +75,7 @@ func (c *tlsHandshakeConn) Read(b []byte) (n int, err error) {
c.packetPending = false
err = c.buf.FinishPacket()
if err != nil {
err = fmt.Errorf("Cannot send handshake packet: %s", err.Error())
err = fmt.Errorf("cannot send handshake packet: %s", err.Error())
return
}
c.continueRead = false
@@ -84,7 +84,7 @@ func (c *tlsHandshakeConn) Read(b []byte) (n int, err error) {
var packet packetType
packet, err = c.buf.BeginRead()
if err != nil {
err = fmt.Errorf("Cannot read handshake packet: %s", err.Error())
err = fmt.Errorf("cannot read handshake packet: %s", err.Error())
return
}
if packet != packPrelogin {
@@ -105,27 +105,27 @@ func (c *tlsHandshakeConn) Write(b []byte) (n int, err error) {
}
func (c *tlsHandshakeConn) Close() error {
panic("Not implemented")
return c.buf.transport.Close()
}
func (c *tlsHandshakeConn) LocalAddr() net.Addr {
panic("Not implemented")
return nil
}
func (c *tlsHandshakeConn) RemoteAddr() net.Addr {
panic("Not implemented")
return nil
}
func (c *tlsHandshakeConn) SetDeadline(t time.Time) error {
panic("Not implemented")
func (c *tlsHandshakeConn) SetDeadline(_ time.Time) error {
return nil
}
func (c *tlsHandshakeConn) SetReadDeadline(t time.Time) error {
panic("Not implemented")
func (c *tlsHandshakeConn) SetReadDeadline(_ time.Time) error {
return nil
}
func (c *tlsHandshakeConn) SetWriteDeadline(t time.Time) error {
panic("Not implemented")
func (c *tlsHandshakeConn) SetWriteDeadline(_ time.Time) error {
return nil
}
// this connection just delegates all methods to it's wrapped connection
@@ -148,21 +148,21 @@ func (c passthroughConn) Close() error {
}
func (c passthroughConn) LocalAddr() net.Addr {
panic("Not implemented")
return c.c.LocalAddr()
}
func (c passthroughConn) RemoteAddr() net.Addr {
panic("Not implemented")
return c.c.RemoteAddr()
}
func (c passthroughConn) SetDeadline(t time.Time) error {
panic("Not implemented")
return c.c.SetDeadline(t)
}
func (c passthroughConn) SetReadDeadline(t time.Time) error {
panic("Not implemented")
return c.c.SetReadDeadline(t)
}
func (c passthroughConn) SetWriteDeadline(t time.Time) error {
panic("Not implemented")
return c.c.SetWriteDeadline(t)
}

View File

@@ -14,6 +14,7 @@ import (
"time"
"unicode/utf16"
//lint:ignore SA1019 MD4 is used by legacy NTLM
"golang.org/x/crypto/md4"
)
@@ -126,18 +127,6 @@ func createDesKey(bytes, material []byte) {
material[7] = (byte)(bytes[6] << 1)
}
func oddParity(bytes []byte) {
for i := 0; i < len(bytes); i++ {
b := bytes[i]
needsParity := (((b >> 7) ^ (b >> 6) ^ (b >> 5) ^ (b >> 4) ^ (b >> 3) ^ (b >> 2) ^ (b >> 1)) & 0x01) == 0
if needsParity {
bytes[i] = bytes[i] | byte(0x01)
} else {
bytes[i] = bytes[i] & byte(0xfe)
}
}
}
func encryptDes(key []byte, cleartext []byte, ciphertext []byte) {
var desKey [8]byte
createDesKey(key, desKey[:])

View File

@@ -22,12 +22,6 @@ type param struct {
buffer []byte
}
const (
fWithRecomp = 1
fNoMetaData = 2
fReuseMetaData = 4
)
var (
sp_Cursor = procId{1, ""}
sp_CursorOpen = procId{2, ""}

View File

@@ -82,19 +82,20 @@ const (
// https://msdn.microsoft.com/en-us/library/dd304214.aspx
const (
packSQLBatch packetType = 1
packRPCRequest = 3
packReply = 4
packRPCRequest packetType = 3
packReply packetType = 4
// 2.2.1.7 Attention: https://msdn.microsoft.com/en-us/library/dd341449.aspx
// 4.19.2 Out-of-Band Attention Signal: https://msdn.microsoft.com/en-us/library/dd305167.aspx
packAttention = 6
packAttention packetType = 6
packBulkLoadBCP = 7
packTransMgrReq = 14
packNormal = 15
packLogin7 = 16
packSSPIMessage = 17
packPrelogin = 18
packBulkLoadBCP packetType = 7
packFedAuthToken packetType = 8
packTransMgrReq packetType = 14
packNormal packetType = 15
packLogin7 packetType = 16
packSSPIMessage packetType = 17
packPrelogin packetType = 18
)
// prelogin fields
@@ -118,6 +119,17 @@ const (
encryptReq = 3 // Encryption is required.
)
const (
featExtSESSIONRECOVERY byte = 0x01
featExtFEDAUTH byte = 0x02
featExtCOLUMNENCRYPTION byte = 0x04
featExtGLOBALTRANSACTIONS byte = 0x05
featExtAZURESQLSUPPORT byte = 0x08
featExtDATACLASSIFICATION byte = 0x09
featExtUTF8SUPPORT byte = 0x0A
featExtTERMINATOR byte = 0xFF
)
type tdsSession struct {
buf *tdsBuffer
loginAck loginAckStruct
@@ -129,6 +141,7 @@ type tdsSession struct {
log optionalLogger
routedServer string
routedPort uint16
returnStatus *ReturnStatus
}
const (
@@ -155,13 +168,13 @@ func (p keySlice) Less(i, j int) bool { return p[i] < p[j] }
func (p keySlice) Swap(i, j int) { p[i], p[j] = p[j], p[i] }
// http://msdn.microsoft.com/en-us/library/dd357559.aspx
func writePrelogin(w *tdsBuffer, fields map[uint8][]byte) error {
func writePrelogin(packetType packetType, w *tdsBuffer, fields map[uint8][]byte) error {
var err error
w.BeginPacket(packPrelogin, false)
w.BeginPacket(packetType, false)
offset := uint16(5*len(fields) + 1)
keys := make(keySlice, 0, len(fields))
for k, _ := range fields {
for k := range fields {
keys = append(keys, k)
}
sort.Sort(keys)
@@ -210,12 +223,15 @@ func readPrelogin(r *tdsBuffer) (map[uint8][]byte, error) {
if err != nil {
return nil, err
}
if packet_type != 4 {
return nil, errors.New("Invalid respones, expected packet type 4, PRELOGIN RESPONSE")
if packet_type != packReply {
return nil, errors.New("invalid respones, expected packet type 4, PRELOGIN RESPONSE")
}
if len(struct_buf) == 0 {
return nil, errors.New("invalid empty PRELOGIN response, it must contain at least one byte")
}
offset := 0
results := map[uint8][]byte{}
for true {
for {
rec_type := struct_buf[offset]
if rec_type == preloginTERMINATOR {
break
@@ -240,6 +256,16 @@ const (
fIntSecurity = 0x80
)
// OptionFlags3
// http://msdn.microsoft.com/en-us/library/dd304019.aspx
const (
fChangePassword = 1
fSendYukonBinaryXML = 2
fUserInstance = 4
fUnknownCollationHandling = 8
fExtension = 0x10
)
// TypeFlags
const (
// 4 bits for fSQLType
@@ -247,12 +273,6 @@ const (
fReadOnlyIntent = 32
)
// OptionFlags3
// https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-tds/773a62b6-ee89-4c02-9e5e-344882630aac
const (
fExtension = 0x10
)
type login struct {
TDSVersion uint32
PacketSize uint32
@@ -295,7 +315,7 @@ func (e *featureExts) Add(f featureExt) error {
}
id := f.featureID()
if _, exists := e.features[id]; exists {
f := "Login error: Feature with ID '%v' is already present in FeatureExt block."
f := "login error: Feature with ID '%v' is already present in FeatureExt block"
return fmt.Errorf(f, id)
}
if e.features == nil {
@@ -326,37 +346,63 @@ func (e featureExts) toBytes() []byte {
return d
}
type featureExtFedAuthSTS struct {
FedAuthEcho bool
// featureExtFedAuth tracks federated authentication state before and during login
type featureExtFedAuth struct {
// FedAuthLibrary is populated by the federated authentication provider.
FedAuthLibrary int
// ADALWorkflow is populated by the federated authentication provider.
ADALWorkflow byte
// FedAuthEcho is populated from the prelogin response
FedAuthEcho bool
// FedAuthToken is populated during login with the value from the provider.
FedAuthToken string
Nonce []byte
// Nonce is populated during login with the value from the provider.
Nonce []byte
// Signature is populated during login with the value from the server.
Signature []byte
}
func (e *featureExtFedAuthSTS) featureID() byte {
return 0x02
func (e *featureExtFedAuth) featureID() byte {
return featExtFEDAUTH
}
func (e *featureExtFedAuthSTS) toBytes() []byte {
func (e *featureExtFedAuth) toBytes() []byte {
if e == nil {
return nil
}
options := byte(0x01) << 1 // 0x01 => STS bFedAuthLibrary 7BIT
options := byte(e.FedAuthLibrary) << 1
if e.FedAuthEcho {
options |= 1 // fFedAuthEcho
}
d := make([]byte, 5)
d[0] = options
// Feature extension format depends on the federated auth library.
// Options are described at
// https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-tds/773a62b6-ee89-4c02-9e5e-344882630aac
var d []byte
// looks like string in
// https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-tds/f88b63bb-b479-49e1-a87b-deda521da508
tokenBytes := str2ucs2(e.FedAuthToken)
binary.LittleEndian.PutUint32(d[1:], uint32(len(tokenBytes))) // Should be a signed int32, but since the length is relatively small, this should work
d = append(d, tokenBytes...)
switch e.FedAuthLibrary {
case fedAuthLibrarySecurityToken:
d = make([]byte, 5)
d[0] = options
if len(e.Nonce) == 32 {
d = append(d, e.Nonce...)
// looks like string in
// https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-tds/f88b63bb-b479-49e1-a87b-deda521da508
tokenBytes := str2ucs2(e.FedAuthToken)
binary.LittleEndian.PutUint32(d[1:], uint32(len(tokenBytes))) // Should be a signed int32, but since the length is relatively small, this should work
d = append(d, tokenBytes...)
if len(e.Nonce) == 32 {
d = append(d, e.Nonce...)
}
case fedAuthLibraryADAL:
d = []byte{options, e.ADALWorkflow}
}
return d
@@ -418,7 +464,7 @@ func str2ucs2(s string) []byte {
func ucs22str(s []byte) (string, error) {
if len(s)%2 != 0 {
return "", fmt.Errorf("Illegal UCS2 string length: %d", len(s))
return "", fmt.Errorf("illegal UCS2 string length: %d", len(s))
}
buf := make([]uint16, len(s)/2)
for i := 0; i < len(s); i += 2 {
@@ -436,7 +482,7 @@ func manglePassword(password string) []byte {
}
// http://msdn.microsoft.com/en-us/library/dd304019.aspx
func sendLogin(w *tdsBuffer, login login) error {
func sendLogin(w *tdsBuffer, login *login) error {
w.BeginPacket(packLogin7, false)
hostname := str2ucs2(login.HostName)
username := str2ucs2(login.UserName)
@@ -572,6 +618,36 @@ func sendLogin(w *tdsBuffer, login login) error {
return w.FinishPacket()
}
// https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-tds/827d9632-2957-4d54-b9ea-384530ae79d0
func sendFedAuthInfo(w *tdsBuffer, fedAuth *featureExtFedAuth) (err error) {
fedauthtoken := str2ucs2(fedAuth.FedAuthToken)
tokenlen := len(fedauthtoken)
datalen := 4 + tokenlen + len(fedAuth.Nonce)
w.BeginPacket(packFedAuthToken, false)
err = binary.Write(w, binary.LittleEndian, uint32(datalen))
if err != nil {
return
}
err = binary.Write(w, binary.LittleEndian, uint32(tokenlen))
if err != nil {
return
}
_, err = w.Write(fedauthtoken)
if err != nil {
return
}
_, err = w.Write(fedAuth.Nonce)
if err != nil {
return
}
return w.FinishPacket()
}
func readUcs2(r io.Reader, numchars int) (res string, err error) {
buf := make([]byte, numchars*2)
_, err = io.ReadFull(r, buf)
@@ -770,12 +846,13 @@ type auth interface {
// use the first one that allows a connection.
func dialConnection(ctx context.Context, c *Connector, p connectParams) (conn net.Conn, err error) {
var ips []net.IP
ips, err = net.LookupIP(p.host)
if err != nil {
ip := net.ParseIP(p.host)
if ip == nil {
return nil, err
ip := net.ParseIP(p.host)
if ip == nil {
ips, err = net.LookupIP(p.host)
if err != nil {
return
}
} else {
ips = []net.IP{ip}
}
if len(ips) == 1 {
@@ -802,7 +879,7 @@ func dialConnection(ctx context.Context, c *Connector, p connectParams) (conn ne
}
// Wait for either the *first* successful connection, or all the errors
wait_loop:
for i, _ := range ips {
for i := range ips {
select {
case conn = <-connChan:
// Got a connection to use, close any others
@@ -824,12 +901,123 @@ func dialConnection(ctx context.Context, c *Connector, p connectParams) (conn ne
}
// Can't do the usual err != nil check, as it is possible to have gotten an error before a successful connection
if conn == nil {
f := "Unable to open tcp connection with host '%v:%v': %v"
f := "unable to open tcp connection with host '%v:%v': %v"
return nil, fmt.Errorf(f, p.host, resolveServerPort(p.port), err.Error())
}
return conn, err
}
func preparePreloginFields(p connectParams, fe *featureExtFedAuth) map[uint8][]byte {
instance_buf := []byte(p.instance)
instance_buf = append(instance_buf, 0) // zero terminate instance name
var encrypt byte
if p.disableEncryption {
encrypt = encryptNotSup
} else if p.encrypt {
encrypt = encryptOn
} else {
encrypt = encryptOff
}
fields := map[uint8][]byte{
preloginVERSION: {0, 0, 0, 0, 0, 0},
preloginENCRYPTION: {encrypt},
preloginINSTOPT: instance_buf,
preloginTHREADID: {0, 0, 0, 0},
preloginMARS: {0}, // MARS disabled
}
if fe.FedAuthLibrary != fedAuthLibraryReserved {
fields[preloginFEDAUTHREQUIRED] = []byte{1}
}
return fields
}
func interpretPreloginResponse(p connectParams, fe *featureExtFedAuth, fields map[uint8][]byte) (encrypt byte, err error) {
// If the server returns the preloginFEDAUTHREQUIRED field, then federated authentication
// is supported. The actual value may be 0 or 1, where 0 means either SSPI or federated
// authentication is allowed, while 1 means only federated authentication is allowed.
if fedAuthSupport, ok := fields[preloginFEDAUTHREQUIRED]; ok {
if len(fedAuthSupport) != 1 {
return 0, fmt.Errorf("Federated authentication flag length should be 1: is %d", len(fedAuthSupport))
}
// We need to be able to echo the value back to the server
fe.FedAuthEcho = fedAuthSupport[0] != 0
} else if fe.FedAuthLibrary != fedAuthLibraryReserved {
return 0, fmt.Errorf("Federated authentication is not supported by the server")
}
encryptBytes, ok := fields[preloginENCRYPTION]
if !ok {
return 0, fmt.Errorf("encrypt negotiation failed")
}
encrypt = encryptBytes[0]
if p.encrypt && (encrypt == encryptNotSup || encrypt == encryptOff) {
return 0, fmt.Errorf("server does not support encryption")
}
return
}
func prepareLogin(ctx context.Context, c *Connector, p connectParams, log optionalLogger, auth auth, fe *featureExtFedAuth, packetSize uint32) (l *login, err error) {
l = &login{
TDSVersion: verTDS74,
PacketSize: packetSize,
Database: p.database,
OptionFlags2: fODBC, // to get unlimited TEXTSIZE
HostName: p.workstation,
ServerName: p.host,
AppName: p.appname,
TypeFlags: p.typeFlags,
}
switch {
case fe.FedAuthLibrary == fedAuthLibrarySecurityToken:
if p.logFlags&logDebug != 0 {
log.Println("Starting federated authentication using security token")
}
fe.FedAuthToken, err = c.securityTokenProvider(ctx)
if err != nil {
if p.logFlags&logDebug != 0 {
log.Printf("Failed to retrieve service principal token for federated authentication security token library: %v", err)
}
return nil, err
}
l.FeatureExt.Add(fe)
case fe.FedAuthLibrary == fedAuthLibraryADAL:
if p.logFlags&logDebug != 0 {
log.Println("Starting federated authentication using ADAL")
}
l.FeatureExt.Add(fe)
case auth != nil:
if p.logFlags&logDebug != 0 {
log.Println("Starting SSPI login")
}
l.SSPI, err = auth.InitialBytes()
if err != nil {
return nil, err
}
l.OptionFlags2 |= fIntSecurity
return l, nil
default:
// Default to SQL server authentication with user and password
l.UserName = p.user
l.Password = p.password
}
return l, nil
}
func connect(ctx context.Context, c *Connector, log optionalLogger, p connectParams) (res *tdsSession, err error) {
dialCtx := ctx
if p.dial_timeout > 0 {
@@ -842,24 +1030,24 @@ func connect(ctx context.Context, c *Connector, log optionalLogger, p connectPar
// both instance name and port specified
// when port is specified instance name is not used
// you should not provide instance name when you provide port
log.Println("WARN: You specified both instance name and port in the connection string, port will be used and instance name will be ignored");
log.Println("WARN: You specified both instance name and port in the connection string, port will be used and instance name will be ignored")
}
if p.instance != "" && p.port == 0 {
p.instance = strings.ToUpper(p.instance)
d := c.getDialer(&p)
instances, err := getInstances(dialCtx, d, p.host)
if err != nil {
f := "Unable to get instances from Sql Server Browser on host %v: %v"
f := "unable to get instances from Sql Server Browser on host %v: %v"
return nil, fmt.Errorf(f, p.host, err.Error())
}
strport, ok := instances[p.instance]["tcp"]
if !ok {
f := "No instance matching '%v' returned from host '%v'"
f := "no instance matching '%v' returned from host '%v'"
return nil, fmt.Errorf(f, p.instance, p.host)
}
port, err := strconv.ParseUint(strport, 0, 16)
if err != nil {
f := "Invalid tcp port returned from Sql Server Browser '%v': %v"
f := "invalid tcp port returned from Sql Server Browser '%v': %v"
return nil, fmt.Errorf(f, strport, err.Error())
}
p.port = port
@@ -880,25 +1068,14 @@ initiate_connection:
logFlags: p.logFlags,
}
instance_buf := []byte(p.instance)
instance_buf = append(instance_buf, 0) // zero terminate instance name
var encrypt byte
if p.disableEncryption {
encrypt = encryptNotSup
} else if p.encrypt {
encrypt = encryptOn
} else {
encrypt = encryptOff
}
fields := map[uint8][]byte{
preloginVERSION: {0, 0, 0, 0, 0, 0},
preloginENCRYPTION: {encrypt},
preloginINSTOPT: instance_buf,
preloginTHREADID: {0, 0, 0, 0},
preloginMARS: {0}, // MARS disabled
fedAuth := &featureExtFedAuth{
FedAuthLibrary: p.fedAuthLibrary,
ADALWorkflow: p.fedAuthADALWorkflow,
}
err = writePrelogin(outbuf, fields)
fields := preparePreloginFields(p, fedAuth)
err = writePrelogin(packPrelogin, outbuf, fields)
if err != nil {
return nil, err
}
@@ -908,13 +1085,9 @@ initiate_connection:
return nil, err
}
encryptBytes, ok := fields[preloginENCRYPTION]
if !ok {
return nil, fmt.Errorf("Encrypt negotiation failed")
}
encrypt = encryptBytes[0]
if p.encrypt && (encrypt == encryptNotSup || encrypt == encryptOff) {
return nil, fmt.Errorf("Server does not support encryption")
encrypt, err := interpretPreloginResponse(p, fedAuth, fields)
if err != nil {
return nil, err
}
if encrypt != encryptNotSup {
@@ -922,7 +1095,7 @@ initiate_connection:
if p.certificate != "" {
pem, err := ioutil.ReadFile(p.certificate)
if err != nil {
return nil, fmt.Errorf("Cannot read certificate %q: %v", p.certificate, err)
return nil, fmt.Errorf("cannot read certificate %q: %v", p.certificate, err)
}
certs := x509.NewCertPool()
certs.AppendCertsFromPEM(pem)
@@ -954,54 +1127,46 @@ initiate_connection:
}
}
login := login{
TDSVersion: verTDS74,
PacketSize: uint32(outbuf.PackageSize()),
Database: p.database,
OptionFlags2: fODBC, // to get unlimited TEXTSIZE
HostName: p.workstation,
ServerName: p.host,
AppName: p.appname,
TypeFlags: p.typeFlags,
}
auth, authOk := getAuth(p.user, p.password, p.serverSPN, p.workstation)
switch {
case p.fedAuthAccessToken != "": // accesstoken ignores user/password
featurext := &featureExtFedAuthSTS{
FedAuthEcho: len(fields[preloginFEDAUTHREQUIRED]) > 0 && fields[preloginFEDAUTHREQUIRED][0] == 1,
FedAuthToken: p.fedAuthAccessToken,
Nonce: fields[preloginNONCEOPT],
}
login.FeatureExt.Add(featurext)
case authOk:
login.SSPI, err = auth.InitialBytes()
if err != nil {
return nil, err
}
login.OptionFlags2 |= fIntSecurity
if authOk {
defer auth.Free()
default:
login.UserName = p.user
login.Password = p.password
} else {
auth = nil
}
login, err := prepareLogin(ctx, c, p, log, auth, fedAuth, uint32(outbuf.PackageSize()))
if err != nil {
return nil, err
}
err = sendLogin(outbuf, login)
if err != nil {
return nil, err
}
// processing login response
success := false
for {
tokchan := make(chan tokenStruct, 5)
go processResponse(context.Background(), &sess, tokchan, nil)
for tok := range tokchan {
// Loop until a packet containing a login acknowledgement is received.
// SSPI and federated authentication scenarios may require multiple
// packet exchanges to complete the login sequence.
for loginAck := false; !loginAck; {
reader := startReading(&sess, ctx, nil)
for {
tok, err := reader.nextToken()
if err != nil {
return nil, err
}
if tok == nil {
break
}
switch token := tok.(type) {
case sspiMsg:
sspi_msg, err := auth.NextBytes(token)
if err != nil {
return nil, err
}
if sspi_msg != nil && len(sspi_msg) > 0 {
if len(sspi_msg) > 0 {
outbuf.BeginPacket(packSSPIMessage, false)
_, err = outbuf.Write(sspi_msg)
if err != nil {
@@ -1013,23 +1178,41 @@ initiate_connection:
}
sspi_msg = nil
}
// TODO: for Live ID authentication it may be necessary to
// compare fedAuth.Nonce == token.Nonce and keep track of signature
//case fedAuthAckStruct:
//fedAuth.Signature = token.Signature
case fedAuthInfoStruct:
// For ADAL workflows this contains the STS URL and server SPN.
// If received outside of an ADAL workflow, ignore.
if c == nil || c.adalTokenProvider == nil {
continue
}
// Request the AD token given the server SPN and STS URL
fedAuth.FedAuthToken, err = c.adalTokenProvider(ctx, token.ServerSPN, token.STSURL)
if err != nil {
return nil, err
}
// Now need to send the token as a FEDINFO packet
err = sendFedAuthInfo(outbuf, fedAuth)
if err != nil {
return nil, err
}
case loginAckStruct:
success = true
sess.loginAck = token
case error:
return nil, fmt.Errorf("Login error: %s", token.Error())
loginAck = true
case doneStruct:
if token.isError() {
return nil, fmt.Errorf("Login error: %s", token.getError())
return nil, fmt.Errorf("login error: %s", token.getError())
}
goto loginEnd
case error:
return nil, fmt.Errorf("login error: %s", token.Error())
}
}
}
loginEnd:
if !success {
return nil, fmt.Errorf("Login failed")
}
if sess.routedServer != "" {
toconn.Close()
p.host = sess.routedServer
@@ -1041,3 +1224,9 @@ loginEnd:
}
return &sess, nil
}
func (sess *tdsSession) setReturnStatus(status ReturnStatus) {
if sess.returnStatus != nil {
*sess.returnStatus = status
}
}

View File

@@ -6,12 +6,11 @@ import (
"errors"
"fmt"
"io"
"net"
"io/ioutil"
"strconv"
"strings"
)
//go:generate stringer -type token
//go:generate go run golang.org/x/tools/cmd/stringer -type token
type token byte
@@ -29,6 +28,7 @@ const (
tokenNbcRow token = 210 // 0xd2
tokenEnvChange token = 227 // 0xE3
tokenSSPI token = 237 // 0xED
tokenFedAuthInfo token = 238 // 0xEE
tokenDone token = 253 // 0xFD
tokenDoneProc token = 254
tokenDoneInProc token = 255
@@ -70,6 +70,11 @@ const (
envRouting = 20
)
const (
fedAuthInfoSTSURL = 0x01
fedAuthInfoSPN = 0x02
)
// COLMETADATA flags
// https://msdn.microsoft.com/en-us/library/dd357363.aspx
const (
@@ -105,26 +110,6 @@ func (d doneStruct) getError() Error {
type doneInProcStruct doneStruct
var doneFlags2str = map[uint16]string{
doneFinal: "final",
doneMore: "more",
doneError: "error",
doneInxact: "inxact",
doneCount: "count",
doneAttn: "attn",
doneSrvError: "srverror",
}
func doneFlags2Str(flags uint16) string {
strs := make([]string, 0, len(doneFlags2str))
for flag, tag := range doneFlags2str {
if flags&flag != 0 {
strs = append(strs, tag)
}
}
return strings.Join(strs, "|")
}
// ENVCHANGE stream
// http://msdn.microsoft.com/en-us/library/dd303449.aspx
func processEnvChg(sess *tdsSession) {
@@ -380,9 +365,8 @@ func processEnvChg(sess *tdsSession) {
default:
// ignore rest of records because we don't know how to skip those
sess.log.Printf("WARN: Unknown ENVCHANGE record detected with type id = %d\n", envtype)
break
return
}
}
}
@@ -425,6 +409,78 @@ func parseSSPIMsg(r *tdsBuffer) sspiMsg {
return sspiMsg(buf)
}
type fedAuthInfoStruct struct {
STSURL string
ServerSPN string
}
type fedAuthInfoOpt struct {
fedAuthInfoID byte
dataLength, dataOffset uint32
}
func parseFedAuthInfo(r *tdsBuffer) fedAuthInfoStruct {
size := r.uint32()
var STSURL, SPN string
var err error
// Each fedAuthInfoOpt is one byte to indicate the info ID,
// then a four byte offset and a four byte length.
count := r.uint32()
offset := uint32(4)
opts := make([]fedAuthInfoOpt, count)
for i := uint32(0); i < count; i++ {
fedAuthInfoID := r.byte()
dataLength := r.uint32()
dataOffset := r.uint32()
offset += 1 + 4 + 4
opts[i] = fedAuthInfoOpt{
fedAuthInfoID: fedAuthInfoID,
dataLength: dataLength,
dataOffset: dataOffset,
}
}
data := make([]byte, size-offset)
r.ReadFull(data)
for i := uint32(0); i < count; i++ {
if opts[i].dataOffset < offset {
badStreamPanicf("Fed auth info opt stated data offset %d is before data begins in packet at %d",
opts[i].dataOffset, offset)
// returns via panic
}
if opts[i].dataOffset+opts[i].dataLength > size {
badStreamPanicf("Fed auth info opt stated data length %d added to stated offset exceeds size of packet %d",
opts[i].dataOffset+opts[i].dataLength, size)
// returns via panic
}
optData := data[opts[i].dataOffset-offset : opts[i].dataOffset-offset+opts[i].dataLength]
switch opts[i].fedAuthInfoID {
case fedAuthInfoSTSURL:
STSURL, err = ucs22str(optData)
case fedAuthInfoSPN:
SPN, err = ucs22str(optData)
default:
err = fmt.Errorf("Unexpected fed auth info opt ID %d", int(opts[i].fedAuthInfoID))
}
if err != nil {
badStreamPanic(err)
}
}
return fedAuthInfoStruct{
STSURL: STSURL,
ServerSPN: SPN,
}
}
type loginAckStruct struct {
Interface uint8
TDSVersion uint32
@@ -449,19 +505,43 @@ func parseLoginAck(r *tdsBuffer) loginAckStruct {
}
// https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-tds/2eb82f8e-11f0-46dc-b42d-27302fa4701a
func parseFeatureExtAck(r *tdsBuffer) {
// at most 1 featureAck per feature in featureExt
// go-mssqldb will add at most 1 feature, the spec defines 7 different features
for i := 0; i < 8; i++ {
featureID := r.byte() // FeatureID
if featureID == 0xff {
return
type fedAuthAckStruct struct {
Nonce []byte
Signature []byte
}
func parseFeatureExtAck(r *tdsBuffer) map[byte]interface{} {
ack := map[byte]interface{}{}
for feature := r.byte(); feature != featExtTERMINATOR; feature = r.byte() {
length := r.uint32()
switch feature {
case featExtFEDAUTH:
// In theory we need to know the federated authentication library to
// know how to parse, but the alternatives provide compatible structures.
fedAuthAck := fedAuthAckStruct{}
if length >= 32 {
fedAuthAck.Nonce = make([]byte, 32)
r.ReadFull(fedAuthAck.Nonce)
length -= 32
}
if length >= 32 {
fedAuthAck.Signature = make([]byte, 32)
r.ReadFull(fedAuthAck.Signature)
length -= 32
}
ack[feature] = fedAuthAck
}
// Skip unprocessed bytes
if length > 0 {
io.CopyN(ioutil.Discard, r, int64(length))
}
size := r.uint32() // FeatureAckDataLen
d := make([]byte, size)
r.ReadFull(d)
}
panic("parsed more than 7 featureAck's, protocol implementation error?")
return ack
}
// http://msdn.microsoft.com/en-us/library/dd357363.aspx
@@ -579,7 +659,7 @@ func processSingleResponse(sess *tdsSession, ch chan tokenStruct, outs map[strin
}
var columns []columnStruct
errs := make([]Error, 0, 5)
for {
for tokens := 0; ; tokens += 1 {
token := token(sess.buf.byte())
if sess.logFlags&logDebug != 0 {
sess.log.Printf("got token %v", token)
@@ -588,6 +668,9 @@ func processSingleResponse(sess *tdsSession, ch chan tokenStruct, outs map[strin
case tokenSSPI:
ch <- parseSSPIMsg(sess.buf)
return
case tokenFedAuthInfo:
ch <- parseFedAuthInfo(sess.buf)
return
case tokenReturnStatus:
returnStatus := parseReturnStatus(sess.buf)
ch <- returnStatus
@@ -595,7 +678,8 @@ func processSingleResponse(sess *tdsSession, ch chan tokenStruct, outs map[strin
loginAck := parseLoginAck(sess.buf)
ch <- loginAck
case tokenFeatureExtAck:
parseFeatureExtAck(sess.buf)
featureExtAck := parseFeatureExtAck(sess.buf)
ch <- featureExtAck
case tokenOrder:
order := parseOrder(sess.buf)
ch <- order
@@ -670,158 +754,137 @@ func processSingleResponse(sess *tdsSession, ch chan tokenStruct, outs map[strin
}
}
type parseRespIter byte
const (
parseRespIterContinue parseRespIter = iota // Continue parsing current token.
parseRespIterNext // Fetch the next token.
parseRespIterDone // Done with parsing the response.
)
type parseRespState byte
const (
parseRespStateNormal parseRespState = iota // Normal response state.
parseRespStateCancel // Query is canceled, wait for server to confirm.
parseRespStateClosing // Waiting for tokens to come through.
)
type parseResp struct {
sess *tdsSession
ctxDone <-chan struct{}
state parseRespState
cancelError error
type tokenProcessor struct {
tokChan chan tokenStruct
ctx context.Context
sess *tdsSession
outs map[string]interface{}
lastRow []interface{}
rowCount int64
firstError error
}
func (ts *parseResp) sendAttention(ch chan tokenStruct) parseRespIter {
if err := sendAttention(ts.sess.buf); err != nil {
ts.dlogf("failed to send attention signal %v", err)
ch <- err
return parseRespIterDone
}
ts.state = parseRespStateCancel
return parseRespIterContinue
}
func (ts *parseResp) dlog(msg string) {
// logging from goroutine is disabled to prevent
// data race detection from firing
// The race is probably happening when
// test logger changes between tests.
/*if ts.sess.logFlags&logDebug != 0 {
ts.sess.log.Println(msg)
}*/
}
func (ts *parseResp) dlogf(f string, v ...interface{}) {
/*if ts.sess.logFlags&logDebug != 0 {
ts.sess.log.Printf(f, v...)
}*/
}
func (ts *parseResp) iter(ctx context.Context, ch chan tokenStruct, tokChan chan tokenStruct) parseRespIter {
switch ts.state {
default:
panic("unknown state")
case parseRespStateNormal:
select {
case tok, ok := <-tokChan:
if !ok {
ts.dlog("response finished")
return parseRespIterDone
}
if err, ok := tok.(net.Error); ok && err.Timeout() {
ts.cancelError = err
ts.dlog("got timeout error, sending attention signal to server")
return ts.sendAttention(ch)
}
// Pass the token along.
ch <- tok
return parseRespIterContinue
case <-ts.ctxDone:
ts.ctxDone = nil
ts.dlog("got cancel message, sending attention signal to server")
return ts.sendAttention(ch)
}
case parseRespStateCancel: // Read all responses until a DONE or error is received.Auth
select {
case tok, ok := <-tokChan:
if !ok {
ts.dlog("response finished but waiting for attention ack")
return parseRespIterNext
}
switch tok := tok.(type) {
default:
// Ignore all other tokens while waiting.
// The TDS spec says other tokens may arrive after an attention
// signal is sent. Ignore these tokens and continue looking for
// a DONE with attention confirm mark.
case doneStruct:
if tok.Status&doneAttn != 0 {
ts.dlog("got cancellation confirmation from server")
if ts.cancelError != nil {
ch <- ts.cancelError
ts.cancelError = nil
} else {
ch <- ctx.Err()
}
return parseRespIterDone
}
// If an error happens during cancel, pass it along and just stop.
// We are uncertain to receive more tokens.
case error:
ch <- tok
ts.state = parseRespStateClosing
}
return parseRespIterContinue
case <-ts.ctxDone:
ts.ctxDone = nil
ts.state = parseRespStateClosing
return parseRespIterContinue
}
case parseRespStateClosing: // Wait for current token chan to close.
if _, ok := <-tokChan; !ok {
ts.dlog("response finished")
return parseRespIterDone
}
return parseRespIterContinue
}
}
func processResponse(ctx context.Context, sess *tdsSession, ch chan tokenStruct, outs map[string]interface{}) {
ts := &parseResp{
func startReading(sess *tdsSession, ctx context.Context, outs map[string]interface{}) *tokenProcessor {
tokChan := make(chan tokenStruct, 5)
go processSingleResponse(sess, tokChan, outs)
return &tokenProcessor{
tokChan: tokChan,
ctx: ctx,
sess: sess,
ctxDone: ctx.Done(),
outs: outs,
}
defer func() {
// Ensure any remaining error is piped through
// or the query may look like it executed when it actually failed.
if ts.cancelError != nil {
ch <- ts.cancelError
ts.cancelError = nil
}
close(ch)
}()
}
// Loop over multiple responses.
func (t *tokenProcessor) iterateResponse() error {
for {
ts.dlog("initiating response reading")
tokChan := make(chan tokenStruct)
go processSingleResponse(sess, tokChan, outs)
// Loop over multiple tokens in response.
tokensLoop:
for {
switch ts.iter(ctx, ch, tokChan) {
case parseRespIterContinue:
// Nothing, continue to next token.
case parseRespIterNext:
break tokensLoop
case parseRespIterDone:
return
tok, err := t.nextToken()
if err == nil {
if tok == nil {
return t.firstError
} else {
switch token := tok.(type) {
case []columnStruct:
t.sess.columns = token
case []interface{}:
t.lastRow = token
case doneInProcStruct:
if token.Status&doneCount != 0 {
t.rowCount += int64(token.RowCount)
}
case doneStruct:
if token.Status&doneCount != 0 {
t.rowCount += int64(token.RowCount)
}
if token.isError() && t.firstError == nil {
t.firstError = token.getError()
}
case ReturnStatus:
t.sess.setReturnStatus(token)
/*case error:
if resultError == nil {
resultError = token
}*/
}
}
} else {
return err
}
}
}
func (t tokenProcessor) nextToken() (tokenStruct, error) {
// we do this separate non-blocking check on token channel to
// prioritize it over cancellation channel
select {
case tok, more := <-t.tokChan:
err, more := tok.(error)
if more {
// this is an error and not a token
return nil, err
} else {
return tok, nil
}
default:
// there are no tokens on the channel, will need to wait
}
select {
case tok, more := <-t.tokChan:
if more {
err, ok := tok.(error)
if ok {
// this is an error and not a token
return nil, err
} else {
return tok, nil
}
} else {
// completed reading response
return nil, nil
}
case <-t.ctx.Done():
if err := sendAttention(t.sess.buf); err != nil {
// unable to send attention, current connection is bad
// notify caller and close channel
return nil, err
}
// now the server should send cancellation confirmation
// it is possible that we already received full response
// just before we sent cancellation request
// in this case current response would not contain confirmation
// and we would need to read one more response
// first lets finish reading current response and look
// for confirmation in it
if readCancelConfirmation(t.tokChan) {
// we got confirmation in current response
return nil, t.ctx.Err()
}
// we did not get cancellation confirmation in the current response
// read one more response, it must be there
t.tokChan = make(chan tokenStruct, 5)
go processSingleResponse(t.sess, t.tokChan, t.outs)
if readCancelConfirmation(t.tokChan) {
return nil, t.ctx.Err()
}
// we did not get cancellation confirmation, something is not
// right, this connection is not usable anymore
return nil, errors.New("did not get cancellation confirmation from the server")
}
}
func readCancelConfirmation(tokChan chan tokenStruct) bool {
for tok := range tokChan {
switch tok := tok.(type) {
default:
// just skip token
case doneStruct:
if tok.Status&doneAttn != 0 {
// got cancellation confirmation, exit
return true
}
}
}
return false
}

View File

@@ -1,29 +1,24 @@
// Code generated by "stringer -type token"; DO NOT EDIT
// Code generated by "stringer -type token"; DO NOT EDIT.
package mssql
import "fmt"
import "strconv"
const (
_token_name_0 = "tokenReturnStatus"
_token_name_1 = "tokenColMetadata"
_token_name_2 = "tokenOrdertokenErrortokenInfo"
_token_name_3 = "tokenLoginAck"
_token_name_4 = "tokenRowtokenNbcRow"
_token_name_5 = "tokenEnvChange"
_token_name_6 = "tokenSSPI"
_token_name_7 = "tokenDonetokenDoneProctokenDoneInProc"
_token_name_2 = "tokenOrdertokenErrortokenInfotokenReturnValuetokenLoginAcktokenFeatureExtAck"
_token_name_3 = "tokenRowtokenNbcRow"
_token_name_4 = "tokenEnvChange"
_token_name_5 = "tokenSSPItokenFedAuthInfo"
_token_name_6 = "tokenDonetokenDoneProctokenDoneInProc"
)
var (
_token_index_0 = [...]uint8{0, 17}
_token_index_1 = [...]uint8{0, 16}
_token_index_2 = [...]uint8{0, 10, 20, 29}
_token_index_3 = [...]uint8{0, 13}
_token_index_4 = [...]uint8{0, 8, 19}
_token_index_5 = [...]uint8{0, 14}
_token_index_6 = [...]uint8{0, 9}
_token_index_7 = [...]uint8{0, 9, 22, 37}
_token_index_2 = [...]uint8{0, 10, 20, 29, 45, 58, 76}
_token_index_3 = [...]uint8{0, 8, 19}
_token_index_5 = [...]uint8{0, 9, 25}
_token_index_6 = [...]uint8{0, 9, 22, 37}
)
func (i token) String() string {
@@ -32,22 +27,21 @@ func (i token) String() string {
return _token_name_0
case i == 129:
return _token_name_1
case 169 <= i && i <= 171:
case 169 <= i && i <= 174:
i -= 169
return _token_name_2[_token_index_2[i]:_token_index_2[i+1]]
case i == 173:
return _token_name_3
case 209 <= i && i <= 210:
i -= 209
return _token_name_4[_token_index_4[i]:_token_index_4[i+1]]
return _token_name_3[_token_index_3[i]:_token_index_3[i+1]]
case i == 227:
return _token_name_5
case i == 237:
return _token_name_6
return _token_name_4
case 237 <= i && i <= 238:
i -= 237
return _token_name_5[_token_index_5[i]:_token_index_5[i+1]]
case 253 <= i && i <= 255:
i -= 253
return _token_name_7[_token_index_7[i]:_token_index_7[i+1]]
return _token_name_6[_token_index_6[i]:_token_index_6[i+1]]
default:
return fmt.Sprintf("token(%d)", i)
return "token(" + strconv.FormatInt(int64(i), 10) + ")"
}
}

View File

@@ -21,11 +21,11 @@ type isoLevel uint8
const (
isolationUseCurrent isoLevel = 0
isolationReadUncommited = 1
isolationReadCommited = 2
isolationRepeatableRead = 3
isolationSerializable = 4
isolationSnapshot = 5
isolationReadUncommited isoLevel = 1
isolationReadCommited isoLevel = 2
isolationRepeatableRead isoLevel = 3
isolationSerializable isoLevel = 4
isolationSnapshot isoLevel = 5
)
func sendBeginXact(buf *tdsBuffer, headers []headerStruct, isolation isoLevel, name string, resetSession bool) (err error) {

View File

@@ -4,6 +4,7 @@ package mssql
import (
"bytes"
"database/sql"
"encoding/binary"
"errors"
"fmt"
@@ -97,6 +98,9 @@ func (tvp TVP) encode(schema, name string, columnStr []columnStruct, tvpFieldInd
for columnStrIdx, fieldIdx := range tvpFieldIndexes {
field := refStr.Field(fieldIdx)
tvpVal := field.Interface()
if tvp.verifyStandardTypeOnNull(buf, tvpVal) {
continue
}
valOf := reflect.ValueOf(tvpVal)
elemKind := field.Kind()
if elemKind == reflect.Ptr && valOf.IsNil() {
@@ -155,7 +159,7 @@ func (tvp TVP) columnTypes() ([]columnStruct, []int, error) {
defaultValues = append(defaultValues, v.Interface())
continue
}
defaultValues = append(defaultValues, reflect.Zero(field.Type).Interface())
defaultValues = append(defaultValues, tvp.createZeroType(reflect.Zero(field.Type).Interface()))
}
if columnCount-len(tvpFieldIndexes) == columnCount {
@@ -209,19 +213,23 @@ func getSchemeAndName(tvpName string) (string, string, error) {
}
splitVal := strings.Split(tvpName, ".")
if len(splitVal) > 2 {
return "", "", errors.New("wrong tvp name")
return "", "", ErrorObjectName
}
const (
openSquareBrackets = "["
closeSquareBrackets = "]"
)
if len(splitVal) == 2 {
res := make([]string, 2)
for key, value := range splitVal {
tmp := strings.Replace(value, "[", "", -1)
tmp = strings.Replace(tmp, "]", "", -1)
tmp := strings.Replace(value, openSquareBrackets, "", -1)
tmp = strings.Replace(tmp, closeSquareBrackets, "", -1)
res[key] = tmp
}
return res[0], res[1], nil
}
tmp := strings.Replace(splitVal[0], "[", "", -1)
tmp = strings.Replace(tmp, "]", "", -1)
tmp := strings.Replace(splitVal[0], openSquareBrackets, "", -1)
tmp = strings.Replace(tmp, closeSquareBrackets, "", -1)
return "", tmp, nil
}
@@ -229,3 +237,56 @@ func getSchemeAndName(tvpName string) (string, string, error) {
func getCountSQLSeparators(str string) int {
return strings.Count(str, sqlSeparator)
}
// verify types https://golang.org/pkg/database/sql/
func (tvp TVP) createZeroType(fieldVal interface{}) interface{} {
const (
defaultBool = false
defaultFloat64 = float64(0)
defaultInt64 = int64(0)
defaultString = ""
)
switch fieldVal.(type) {
case sql.NullBool:
return defaultBool
case sql.NullFloat64:
return defaultFloat64
case sql.NullInt64:
return defaultInt64
case sql.NullString:
return defaultString
}
return fieldVal
}
// verify types https://golang.org/pkg/database/sql/
func (tvp TVP) verifyStandardTypeOnNull(buf *bytes.Buffer, tvpVal interface{}) bool {
const (
defaultNull = uint8(0)
)
switch val := tvpVal.(type) {
case sql.NullBool:
if !val.Valid {
binary.Write(buf, binary.LittleEndian, defaultNull)
return true
}
case sql.NullFloat64:
if !val.Valid {
binary.Write(buf, binary.LittleEndian, defaultNull)
return true
}
case sql.NullInt64:
if !val.Valid {
binary.Write(buf, binary.LittleEndian, defaultNull)
return true
}
case sql.NullString:
if !val.Valid {
binary.Write(buf, binary.LittleEndian, uint64(_PLP_NULL))
return true
}
}
return false
}

View File

@@ -665,7 +665,7 @@ func readPLPType(ti *typeInfo, r *tdsBuffer) interface{} {
default:
buf = bytes.NewBuffer(make([]byte, 0, size))
}
for true {
for {
chunksize := r.uint32()
if chunksize == 0 {
break
@@ -690,6 +690,10 @@ func readPLPType(ti *typeInfo, r *tdsBuffer) interface{} {
}
func writePLPType(w io.Writer, ti typeInfo, buf []byte) (err error) {
if buf == nil {
err = binary.Write(w, binary.LittleEndian, uint64(_PLP_NULL))
return
}
if err = binary.Write(w, binary.LittleEndian, uint64(_UNKNOWN_PLP_LEN)); err != nil {
return
}
@@ -807,7 +811,6 @@ func readVarLen(ti *typeInfo, r *tdsBuffer) {
default:
badStreamPanicf("Invalid type %d", ti.TypeId)
}
return
}
func decodeMoney(buf []byte) []byte {
@@ -834,8 +837,7 @@ func decodeGuid(buf []byte) []byte {
}
func decodeDecimal(prec uint8, scale uint8, buf []byte) []byte {
var sign uint8
sign = buf[0]
sign := buf[0]
var dec decimal.Decimal
dec.SetPositive(sign != 0)
dec.SetPrec(prec)
@@ -1187,7 +1189,7 @@ func makeDecl(ti typeInfo) string {
return fmt.Sprintf("char(%d)", ti.Size)
case typeBigVarChar, typeVarChar:
if ti.Size > 8000 || ti.Size == 0 {
return fmt.Sprintf("varchar(max)")
return "varchar(max)"
} else {
return fmt.Sprintf("varchar(%d)", ti.Size)
}

View File

@@ -1,13 +0,0 @@
language: go
script:
- go vet ./...
- go test -v ./...
go:
- 1.3
- 1.4
- 1.5
- 1.6
- 1.7
- tip

View File

@@ -1,97 +0,0 @@
## Migration Guide from v2 -> v3
Version 3 adds several new, frequently requested features. To do so, it introduces a few breaking changes. We've worked to keep these as minimal as possible. This guide explains the breaking changes and how you can quickly update your code.
### `Token.Claims` is now an interface type
The most requested feature from the 2.0 verison of this library was the ability to provide a custom type to the JSON parser for claims. This was implemented by introducing a new interface, `Claims`, to replace `map[string]interface{}`. We also included two concrete implementations of `Claims`: `MapClaims` and `StandardClaims`.
`MapClaims` is an alias for `map[string]interface{}` with built in validation behavior. It is the default claims type when using `Parse`. The usage is unchanged except you must type cast the claims property.
The old example for parsing a token looked like this..
```go
if token, err := jwt.Parse(tokenString, keyLookupFunc); err == nil {
fmt.Printf("Token for user %v expires %v", token.Claims["user"], token.Claims["exp"])
}
```
is now directly mapped to...
```go
if token, err := jwt.Parse(tokenString, keyLookupFunc); err == nil {
claims := token.Claims.(jwt.MapClaims)
fmt.Printf("Token for user %v expires %v", claims["user"], claims["exp"])
}
```
`StandardClaims` is designed to be embedded in your custom type. You can supply a custom claims type with the new `ParseWithClaims` function. Here's an example of using a custom claims type.
```go
type MyCustomClaims struct {
User string
*StandardClaims
}
if token, err := jwt.ParseWithClaims(tokenString, &MyCustomClaims{}, keyLookupFunc); err == nil {
claims := token.Claims.(*MyCustomClaims)
fmt.Printf("Token for user %v expires %v", claims.User, claims.StandardClaims.ExpiresAt)
}
```
### `ParseFromRequest` has been moved
To keep this library focused on the tokens without becoming overburdened with complex request processing logic, `ParseFromRequest` and its new companion `ParseFromRequestWithClaims` have been moved to a subpackage, `request`. The method signatues have also been augmented to receive a new argument: `Extractor`.
`Extractors` do the work of picking the token string out of a request. The interface is simple and composable.
This simple parsing example:
```go
if token, err := jwt.ParseFromRequest(tokenString, req, keyLookupFunc); err == nil {
fmt.Printf("Token for user %v expires %v", token.Claims["user"], token.Claims["exp"])
}
```
is directly mapped to:
```go
if token, err := request.ParseFromRequest(req, request.OAuth2Extractor, keyLookupFunc); err == nil {
claims := token.Claims.(jwt.MapClaims)
fmt.Printf("Token for user %v expires %v", claims["user"], claims["exp"])
}
```
There are several concrete `Extractor` types provided for your convenience:
* `HeaderExtractor` will search a list of headers until one contains content.
* `ArgumentExtractor` will search a list of keys in request query and form arguments until one contains content.
* `MultiExtractor` will try a list of `Extractors` in order until one returns content.
* `AuthorizationHeaderExtractor` will look in the `Authorization` header for a `Bearer` token.
* `OAuth2Extractor` searches the places an OAuth2 token would be specified (per the spec): `Authorization` header and `access_token` argument
* `PostExtractionFilter` wraps an `Extractor`, allowing you to process the content before it's parsed. A simple example is stripping the `Bearer ` text from a header
### RSA signing methods no longer accept `[]byte` keys
Due to a [critical vulnerability](https://auth0.com/blog/2015/03/31/critical-vulnerabilities-in-json-web-token-libraries/), we've decided the convenience of accepting `[]byte` instead of `rsa.PublicKey` or `rsa.PrivateKey` isn't worth the risk of misuse.
To replace this behavior, we've added two helper methods: `ParseRSAPrivateKeyFromPEM(key []byte) (*rsa.PrivateKey, error)` and `ParseRSAPublicKeyFromPEM(key []byte) (*rsa.PublicKey, error)`. These are just simple helpers for unpacking PEM encoded PKCS1 and PKCS8 keys. If your keys are encoded any other way, all you need to do is convert them to the `crypto/rsa` package's types.
```go
func keyLookupFunc(*Token) (interface{}, error) {
// Don't forget to validate the alg is what you expect:
if _, ok := token.Method.(*jwt.SigningMethodRSA); !ok {
return nil, fmt.Errorf("Unexpected signing method: %v", token.Header["alg"])
}
// Look up key
key, err := lookupPublicKey(token.Header["kid"])
if err != nil {
return nil, err
}
// Unpack key from PEM encoded PKCS8
return jwt.ParseRSAPublicKeyFromPEM(key)
}
```

View File

@@ -1,129 +0,0 @@
sudo: false
language: go
go:
- 1.10.x
- 1.11.x
- 1.12.x
- 1.13.x
- master
before_install:
- go get golang.org/x/tools/cmd/cover
- go get github.com/mattn/goveralls
before_script:
- echo -e "[server]\ninnodb_log_file_size=256MB\ninnodb_buffer_pool_size=512MB\nmax_allowed_packet=16MB" | sudo tee -a /etc/mysql/my.cnf
- sudo service mysql restart
- .travis/wait_mysql.sh
- mysql -e 'create database gotest;'
matrix:
include:
- env: DB=MYSQL8
sudo: required
dist: trusty
go: 1.10.x
services:
- docker
before_install:
- go get golang.org/x/tools/cmd/cover
- go get github.com/mattn/goveralls
- docker pull mysql:8.0
- docker run -d -p 127.0.0.1:3307:3306 --name mysqld -e MYSQL_DATABASE=gotest -e MYSQL_USER=gotest -e MYSQL_PASSWORD=secret -e MYSQL_ROOT_PASSWORD=verysecret
mysql:8.0 --innodb_log_file_size=256MB --innodb_buffer_pool_size=512MB --max_allowed_packet=16MB --local-infile=1
- cp .travis/docker.cnf ~/.my.cnf
- .travis/wait_mysql.sh
before_script:
- export MYSQL_TEST_USER=gotest
- export MYSQL_TEST_PASS=secret
- export MYSQL_TEST_ADDR=127.0.0.1:3307
- export MYSQL_TEST_CONCURRENT=1
- env: DB=MYSQL57
sudo: required
dist: trusty
go: 1.10.x
services:
- docker
before_install:
- go get golang.org/x/tools/cmd/cover
- go get github.com/mattn/goveralls
- docker pull mysql:5.7
- docker run -d -p 127.0.0.1:3307:3306 --name mysqld -e MYSQL_DATABASE=gotest -e MYSQL_USER=gotest -e MYSQL_PASSWORD=secret -e MYSQL_ROOT_PASSWORD=verysecret
mysql:5.7 --innodb_log_file_size=256MB --innodb_buffer_pool_size=512MB --max_allowed_packet=16MB --local-infile=1
- cp .travis/docker.cnf ~/.my.cnf
- .travis/wait_mysql.sh
before_script:
- export MYSQL_TEST_USER=gotest
- export MYSQL_TEST_PASS=secret
- export MYSQL_TEST_ADDR=127.0.0.1:3307
- export MYSQL_TEST_CONCURRENT=1
- env: DB=MARIA55
sudo: required
dist: trusty
go: 1.10.x
services:
- docker
before_install:
- go get golang.org/x/tools/cmd/cover
- go get github.com/mattn/goveralls
- docker pull mariadb:5.5
- docker run -d -p 127.0.0.1:3307:3306 --name mysqld -e MYSQL_DATABASE=gotest -e MYSQL_USER=gotest -e MYSQL_PASSWORD=secret -e MYSQL_ROOT_PASSWORD=verysecret
mariadb:5.5 --innodb_log_file_size=256MB --innodb_buffer_pool_size=512MB --max_allowed_packet=16MB --local-infile=1
- cp .travis/docker.cnf ~/.my.cnf
- .travis/wait_mysql.sh
before_script:
- export MYSQL_TEST_USER=gotest
- export MYSQL_TEST_PASS=secret
- export MYSQL_TEST_ADDR=127.0.0.1:3307
- export MYSQL_TEST_CONCURRENT=1
- env: DB=MARIA10_1
sudo: required
dist: trusty
go: 1.10.x
services:
- docker
before_install:
- go get golang.org/x/tools/cmd/cover
- go get github.com/mattn/goveralls
- docker pull mariadb:10.1
- docker run -d -p 127.0.0.1:3307:3306 --name mysqld -e MYSQL_DATABASE=gotest -e MYSQL_USER=gotest -e MYSQL_PASSWORD=secret -e MYSQL_ROOT_PASSWORD=verysecret
mariadb:10.1 --innodb_log_file_size=256MB --innodb_buffer_pool_size=512MB --max_allowed_packet=16MB --local-infile=1
- cp .travis/docker.cnf ~/.my.cnf
- .travis/wait_mysql.sh
before_script:
- export MYSQL_TEST_USER=gotest
- export MYSQL_TEST_PASS=secret
- export MYSQL_TEST_ADDR=127.0.0.1:3307
- export MYSQL_TEST_CONCURRENT=1
- os: osx
osx_image: xcode10.1
addons:
homebrew:
packages:
- mysql
update: true
go: 1.12.x
before_install:
- go get golang.org/x/tools/cmd/cover
- go get github.com/mattn/goveralls
before_script:
- echo -e "[server]\ninnodb_log_file_size=256MB\ninnodb_buffer_pool_size=512MB\nmax_allowed_packet=16MB\nlocal_infile=1" >> /usr/local/etc/my.cnf
- mysql.server start
- mysql -uroot -e 'CREATE USER gotest IDENTIFIED BY "secret"'
- mysql -uroot -e 'GRANT ALL ON *.* TO gotest'
- mysql -uroot -e 'create database gotest;'
- export MYSQL_TEST_USER=gotest
- export MYSQL_TEST_PASS=secret
- export MYSQL_TEST_ADDR=127.0.0.1:3306
- export MYSQL_TEST_CONCURRENT=1
script:
- go test -v -covermode=count -coverprofile=coverage.out
- go vet ./...
- .travis/gofmt.sh
after_script:
- $HOME/gopath/bin/goveralls -coverprofile=coverage.out -service=travis-ci

View File

@@ -13,11 +13,15 @@
Aaron Hopkins <go-sql-driver at die.net>
Achille Roussel <achille.roussel at gmail.com>
Alex Snast <alexsn at fb.com>
Alexey Palazhchenko <alexey.palazhchenko at gmail.com>
Andrew Reid <andrew.reid at tixtrack.com>
Animesh Ray <mail.rayanimesh at gmail.com>
Arne Hormann <arnehormann at gmail.com>
Ariel Mashraki <ariel at mashraki.co.il>
Asta Xie <xiemengjun at gmail.com>
Bulat Gaifullin <gaifullinbf at gmail.com>
Caine Jette <jette at alum.mit.edu>
Carlos Nieto <jose.carlos at menteslibres.net>
Chris Moos <chris at tech9computers.com>
Craig Wilson <craiggwilson at gmail.com>
@@ -52,6 +56,7 @@ Julien Schmidt <go-sql-driver at julienschmidt.com>
Justin Li <jli at j-li.net>
Justin Nuß <nuss.justin at gmail.com>
Kamil Dziedzic <kamil at klecza.pl>
Kei Kamikawa <x00.x7f.x86 at gmail.com>
Kevin Malachowski <kevin at chowski.com>
Kieron Woodhouse <kieron.woodhouse at infosum.com>
Lennart Rudolph <lrudolph at hmc.edu>
@@ -74,20 +79,26 @@ Reed Allman <rdallman10 at gmail.com>
Richard Wilkes <wilkes at me.com>
Robert Russell <robert at rrbrussell.com>
Runrioter Wung <runrioter at gmail.com>
Sho Iizuka <sho.i518 at gmail.com>
Sho Ikeda <suicaicoca at gmail.com>
Shuode Li <elemount at qq.com>
Simon J Mudd <sjmudd at pobox.com>
Soroush Pour <me at soroushjp.com>
Stan Putrya <root.vagner at gmail.com>
Stanley Gunawan <gunawan.stanley at gmail.com>
Steven Hartland <steven.hartland at multiplay.co.uk>
Tan Jinhua <312841925 at qq.com>
Thomas Wodarek <wodarekwebpage at gmail.com>
Tim Ruffles <timruffles at gmail.com>
Tom Jenkinson <tom at tjenkinson.me>
Vladimir Kovpak <cn007b at gmail.com>
Vladyslav Zhelezniak <zhvladi at gmail.com>
Xiangyu Hu <xiangyu.hu at outlook.com>
Xiaobing Jiang <s7v7nislands at gmail.com>
Xiuming Chen <cc at cxm.cc>
Xuehong Chan <chanxuehong at gmail.com>
Zhenye Xie <xiezhenye at gmail.com>
Zhixin Wen <john.wenzhixin at gmail.com>
# Organizations
@@ -103,3 +114,4 @@ Multiplay Ltd.
Percona LLC
Pivotal Inc.
Stripe Inc.
Zendesk Inc.

View File

@@ -1,3 +1,29 @@
## Version 1.6 (2021-04-01)
Changes:
- Migrate the CI service from travis-ci to GitHub Actions (#1176, #1183, #1190)
- `NullTime` is deprecated (#960, #1144)
- Reduce allocations when building SET command (#1111)
- Performance improvement for time formatting (#1118)
- Performance improvement for time parsing (#1098, #1113)
New Features:
- Implement `driver.Validator` interface (#1106, #1174)
- Support returning `uint64` from `Valuer` in `ConvertValue` (#1143)
- Add `json.RawMessage` for converter and prepared statement (#1059)
- Interpolate `json.RawMessage` as `string` (#1058)
- Implements `CheckNamedValue` (#1090)
Bugfixes:
- Stop rounding times (#1121, #1172)
- Put zero filler into the SSL handshake packet (#1066)
- Fix checking cancelled connections back into the connection pool (#1095)
- Fix remove last 0 byte for mysql_old_password when password is empty (#1133)
## Version 1.5 (2020-01-07)
Changes:

View File

@@ -35,7 +35,7 @@ A MySQL-Driver for Go's [database/sql](https://golang.org/pkg/database/sql/) pac
* Supports queries larger than 16MB
* Full [`sql.RawBytes`](https://golang.org/pkg/database/sql/#RawBytes) support.
* Intelligent `LONG DATA` handling in prepared statements
* Secure `LOAD DATA LOCAL INFILE` support with file Whitelisting and `io.Reader` support
* Secure `LOAD DATA LOCAL INFILE` support with file allowlisting and `io.Reader` support
* Optional `time.Time` parsing
* Optional placeholder interpolation
@@ -56,15 +56,37 @@ Make sure [Git is installed](https://git-scm.com/downloads) on your machine and
_Go MySQL Driver_ is an implementation of Go's `database/sql/driver` interface. You only need to import the driver and can use the full [`database/sql`](https://golang.org/pkg/database/sql/) API then.
Use `mysql` as `driverName` and a valid [DSN](#dsn-data-source-name) as `dataSourceName`:
```go
import "database/sql"
import _ "github.com/go-sql-driver/mysql"
import (
"database/sql"
"time"
_ "github.com/go-sql-driver/mysql"
)
// ...
db, err := sql.Open("mysql", "user:password@/dbname")
if err != nil {
panic(err)
}
// See "Important settings" section.
db.SetConnMaxLifetime(time.Minute * 3)
db.SetMaxOpenConns(10)
db.SetMaxIdleConns(10)
```
[Examples are available in our Wiki](https://github.com/go-sql-driver/mysql/wiki/Examples "Go-MySQL-Driver Examples").
### Important settings
`db.SetConnMaxLifetime()` is required to ensure connections are closed by the driver safely before connection is closed by MySQL server, OS, or other middlewares. Since some middlewares close idle connections by 5 minutes, we recommend timeout shorter than 5 minutes. This setting helps load balancing and changing system variables too.
`db.SetMaxOpenConns()` is highly recommended to limit the number of connection used by the application. There is no recommended limit number because it depends on application and MySQL server.
`db.SetMaxIdleConns()` is recommended to be set same to (or greater than) `db.SetMaxOpenConns()`. When it is smaller than `SetMaxOpenConns()`, connections can be opened and closed very frequently than you expect. Idle connections can be closed by the `db.SetConnMaxLifetime()`. If you want to close idle connections more rapidly, you can use `db.SetConnMaxIdleTime()` since Go 1.15.
### DSN (Data Source Name)
@@ -122,7 +144,7 @@ Valid Values: true, false
Default: false
```
`allowAllFiles=true` disables the file Whitelist for `LOAD DATA LOCAL INFILE` and allows *all* files.
`allowAllFiles=true` disables the file allowlist for `LOAD DATA LOCAL INFILE` and allows *all* files.
[*Might be insecure!*](http://dev.mysql.com/doc/refman/5.7/en/load-data-local.html)
##### `allowCleartextPasswords`
@@ -133,7 +155,7 @@ Valid Values: true, false
Default: false
```
`allowCleartextPasswords=true` allows using the [cleartext client side plugin](http://dev.mysql.com/doc/en/cleartext-authentication-plugin.html) if required by an account, such as one defined with the [PAM authentication plugin](http://dev.mysql.com/doc/en/pam-authentication-plugin.html). Sending passwords in clear text may be a security problem in some configurations. To avoid problems if there is any possibility that the password would be intercepted, clients should connect to MySQL Server using a method that protects the password. Possibilities include [TLS / SSL](#tls), IPsec, or a private network.
`allowCleartextPasswords=true` allows using the [cleartext client side plugin](https://dev.mysql.com/doc/en/cleartext-pluggable-authentication.html) if required by an account, such as one defined with the [PAM authentication plugin](http://dev.mysql.com/doc/en/pam-authentication-plugin.html). Sending passwords in clear text may be a security problem in some configurations. To avoid problems if there is any possibility that the password would be intercepted, clients should connect to MySQL Server using a method that protects the password. Possibilities include [TLS / SSL](#tls), IPsec, or a private network.
##### `allowNativePasswords`
@@ -230,7 +252,7 @@ Default: false
If `interpolateParams` is true, placeholders (`?`) in calls to `db.Query()` and `db.Exec()` are interpolated into a single query string with given parameters. This reduces the number of roundtrips, since the driver has to prepare a statement, execute it with given parameters and close the statement again with `interpolateParams=false`.
*This can not be used together with the multibyte encodings BIG5, CP932, GB2312, GBK or SJIS. These are blacklisted as they may [introduce a SQL injection vulnerability](http://stackoverflow.com/a/12118602/3430118)!*
*This can not be used together with the multibyte encodings BIG5, CP932, GB2312, GBK or SJIS. These are rejected as they may [introduce a SQL injection vulnerability](http://stackoverflow.com/a/12118602/3430118)!*
##### `loc`
@@ -376,7 +398,7 @@ Rules:
Examples:
* `autocommit=1`: `SET autocommit=1`
* [`time_zone=%27Europe%2FParis%27`](https://dev.mysql.com/doc/refman/5.5/en/time-zone-support.html): `SET time_zone='Europe/Paris'`
* [`tx_isolation=%27REPEATABLE-READ%27`](https://dev.mysql.com/doc/refman/5.5/en/server-system-variables.html#sysvar_tx_isolation): `SET tx_isolation='REPEATABLE-READ'`
* [`transaction_isolation=%27REPEATABLE-READ%27`](https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_transaction_isolation): `SET transaction_isolation='REPEATABLE-READ'`
#### Examples
@@ -445,7 +467,7 @@ For this feature you need direct access to the package. Therefore you must chang
import "github.com/go-sql-driver/mysql"
```
Files must be whitelisted by registering them with `mysql.RegisterLocalFile(filepath)` (recommended) or the Whitelist check must be deactivated by using the DSN parameter `allowAllFiles=true` ([*Might be insecure!*](http://dev.mysql.com/doc/refman/5.7/en/load-data-local.html)).
Files must be explicitly allowed by registering them with `mysql.RegisterLocalFile(filepath)` (recommended) or the allowlist check must be deactivated by using the DSN parameter `allowAllFiles=true` ([*Might be insecure!*](http://dev.mysql.com/doc/refman/5.7/en/load-data-local.html)).
To use a `io.Reader` a handler function must be registered with `mysql.RegisterReaderHandler(name, handler)` which returns a `io.Reader` or `io.ReadCloser`. The Reader is available with the filepath `Reader::<name>` then. Choose different names for different handlers and `DeregisterReaderHandler` when you don't need it anymore.
@@ -459,8 +481,6 @@ However, many want to scan MySQL `DATE` and `DATETIME` values into `time.Time` v
**Caution:** As of Go 1.1, this makes `time.Time` the only variable type you can scan `DATE` and `DATETIME` values into. This breaks for example [`sql.RawBytes` support](https://github.com/go-sql-driver/mysql/wiki/Examples#rawbytes).
Alternatively you can use the [`NullTime`](https://godoc.org/github.com/go-sql-driver/mysql#NullTime) type as the scan destination, which works with both `time.Time` and `string` / `[]byte`.
### Unicode support
Since version 1.5 Go-MySQL-Driver automatically uses the collation ` utf8mb4_general_ci` by default.
@@ -477,7 +497,7 @@ To run the driver tests you may need to adjust the configuration. See the [Testi
Go-MySQL-Driver is not feature-complete yet. Your help is very appreciated.
If you want to contribute, you can work on an [open issue](https://github.com/go-sql-driver/mysql/issues?state=open) or review a [pull request](https://github.com/go-sql-driver/mysql/pulls).
See the [Contribution Guidelines](https://github.com/go-sql-driver/mysql/blob/master/CONTRIBUTING.md) for details.
See the [Contribution Guidelines](https://github.com/go-sql-driver/mysql/blob/master/.github/CONTRIBUTING.md) for details.
---------------------------------------
@@ -498,4 +518,3 @@ Please read the [MPL 2.0 FAQ](https://www.mozilla.org/en-US/MPL/2.0/FAQ/) if you
You can read the full terms here: [LICENSE](https://raw.github.com/go-sql-driver/mysql/master/LICENSE).
![Go Gopher and MySQL Dolphin](https://raw.github.com/wiki/go-sql-driver/mysql/go-mysql-driver_m.jpg "Golang Gopher transporting the MySQL Dolphin in a wheelbarrow")

View File

@@ -15,6 +15,7 @@ import (
"crypto/sha256"
"crypto/x509"
"encoding/pem"
"fmt"
"sync"
)
@@ -136,10 +137,6 @@ func pwHash(password []byte) (result [2]uint32) {
// Hash password using insecure pre 4.1 method
func scrambleOldPassword(scramble []byte, password string) []byte {
if len(password) == 0 {
return nil
}
scramble = scramble[:8]
hashPw := pwHash([]byte(password))
@@ -247,6 +244,9 @@ func (mc *mysqlConn) auth(authData []byte, plugin string) ([]byte, error) {
if !mc.cfg.AllowOldPasswords {
return nil, ErrOldPassword
}
if len(mc.cfg.Passwd) == 0 {
return nil, nil
}
// Note: there are edge cases where this should work but doesn't;
// this is currently "wontfix":
// https://github.com/go-sql-driver/mysql/issues/184
@@ -372,7 +372,10 @@ func (mc *mysqlConn) handleAuthResult(oldAuthData []byte, plugin string) error {
return err
}
block, _ := pem.Decode(data[1:])
block, rest := pem.Decode(data[1:])
if block == nil {
return fmt.Errorf("No Pem data found, data: %s", rest)
}
pkix, err := x509.ParsePKIXPublicKey(block.Bytes)
if err != nil {
return err

View File

@@ -247,7 +247,7 @@ var collations = map[string]byte{
"utf8mb4_0900_ai_ci": 255,
}
// A blacklist of collations which is unsafe to interpolate parameters.
// A denylist of collations which is unsafe to interpolate parameters.
// These multibyte encodings may contains 0x5c (`\`) in their trailing bytes.
var unsafeCollations = map[string]bool{
"big5_chinese_ci": true,

View File

@@ -12,6 +12,7 @@ import (
"context"
"database/sql"
"database/sql/driver"
"encoding/json"
"io"
"net"
"strconv"
@@ -46,9 +47,10 @@ type mysqlConn struct {
// Handles parameters set in DSN after the connection is established
func (mc *mysqlConn) handleParams() (err error) {
var cmdSet strings.Builder
for param, val := range mc.cfg.Params {
switch param {
// Charset
// Charset: character_set_connection, character_set_client, character_set_results
case "charset":
charsets := strings.Split(val, ",")
for i := range charsets {
@@ -62,12 +64,25 @@ func (mc *mysqlConn) handleParams() (err error) {
return
}
// System Vars
// Other system vars accumulated in a single SET command
default:
err = mc.exec("SET " + param + "=" + val + "")
if err != nil {
return
if cmdSet.Len() == 0 {
// Heuristic: 29 chars for each other key=value to reduce reallocations
cmdSet.Grow(4 + len(param) + 1 + len(val) + 30*(len(mc.cfg.Params)-1))
cmdSet.WriteString("SET ")
} else {
cmdSet.WriteByte(',')
}
cmdSet.WriteString(param)
cmdSet.WriteByte('=')
cmdSet.WriteString(val)
}
}
if cmdSet.Len() > 0 {
err = mc.exec(cmdSet.String())
if err != nil {
return
}
}
@@ -230,47 +245,21 @@ func (mc *mysqlConn) interpolateParams(query string, args []driver.Value) (strin
if v.IsZero() {
buf = append(buf, "'0000-00-00'"...)
} else {
v := v.In(mc.cfg.Loc)
v = v.Add(time.Nanosecond * 500) // To round under microsecond
year := v.Year()
year100 := year / 100
year1 := year % 100
month := v.Month()
day := v.Day()
hour := v.Hour()
minute := v.Minute()
second := v.Second()
micro := v.Nanosecond() / 1000
buf = append(buf, []byte{
'\'',
digits10[year100], digits01[year100],
digits10[year1], digits01[year1],
'-',
digits10[month], digits01[month],
'-',
digits10[day], digits01[day],
' ',
digits10[hour], digits01[hour],
':',
digits10[minute], digits01[minute],
':',
digits10[second], digits01[second],
}...)
if micro != 0 {
micro10000 := micro / 10000
micro100 := micro / 100 % 100
micro1 := micro % 100
buf = append(buf, []byte{
'.',
digits10[micro10000], digits01[micro10000],
digits10[micro100], digits01[micro100],
digits10[micro1], digits01[micro1],
}...)
buf = append(buf, '\'')
buf, err = appendDateTime(buf, v.In(mc.cfg.Loc))
if err != nil {
return "", err
}
buf = append(buf, '\'')
}
case json.RawMessage:
buf = append(buf, '\'')
if mc.status&statusNoBackslashEscapes == 0 {
buf = escapeBytesBackslash(buf, v)
} else {
buf = escapeBytesQuotes(buf, v)
}
buf = append(buf, '\'')
case []byte:
if v == nil {
buf = append(buf, "NULL"...)
@@ -480,6 +469,10 @@ func (mc *mysqlConn) Ping(ctx context.Context) (err error) {
// BeginTx implements driver.ConnBeginTx interface
func (mc *mysqlConn) BeginTx(ctx context.Context, opts driver.TxOptions) (driver.Tx, error) {
if mc.closed.IsSet() {
return nil, driver.ErrBadConn
}
if err := mc.watchCancel(ctx); err != nil {
return nil, err
}
@@ -649,3 +642,9 @@ func (mc *mysqlConn) ResetSession(ctx context.Context) error {
mc.reset = true
return nil
}
// IsValid implements driver.Validator interface
// (From Go 1.15)
func (mc *mysqlConn) IsValid() bool {
return !mc.closed.IsSet()
}

View File

@@ -375,7 +375,7 @@ func parseDSNParams(cfg *Config, params string) (err error) {
// cfg params
switch value := param[1]; param[0] {
// Disable INFILE whitelist / enable all files
// Disable INFILE allowlist / enable all files
case "allowAllFiles":
var isBool bool
cfg.AllowAllFiles, isBool = readBool(value)

View File

@@ -106,7 +106,7 @@ var (
scanTypeInt64 = reflect.TypeOf(int64(0))
scanTypeNullFloat = reflect.TypeOf(sql.NullFloat64{})
scanTypeNullInt = reflect.TypeOf(sql.NullInt64{})
scanTypeNullTime = reflect.TypeOf(NullTime{})
scanTypeNullTime = reflect.TypeOf(nullTime{})
scanTypeUint8 = reflect.TypeOf(uint8(0))
scanTypeUint16 = reflect.TypeOf(uint16(0))
scanTypeUint32 = reflect.TypeOf(uint32(0))

24
vendor/github.com/go-sql-driver/mysql/fuzz.go generated vendored Normal file
View File

@@ -0,0 +1,24 @@
// Go MySQL Driver - A MySQL-Driver for Go's database/sql package.
//
// Copyright 2020 The Go-MySQL-Driver Authors. All rights reserved.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// +build gofuzz
package mysql
import (
"database/sql"
)
func Fuzz(data []byte) int {
db, err := sql.Open("mysql", string(data))
if err != nil {
return 0
}
db.Close()
return 1
}

View File

@@ -23,7 +23,7 @@ var (
readerRegisterLock sync.RWMutex
)
// RegisterLocalFile adds the given file to the file whitelist,
// RegisterLocalFile adds the given file to the file allowlist,
// so that it can be used by "LOAD DATA LOCAL INFILE <filepath>".
// Alternatively you can allow the use of all local files with
// the DSN parameter 'allowAllFiles=true'
@@ -45,7 +45,7 @@ func RegisterLocalFile(filePath string) {
fileRegisterLock.Unlock()
}
// DeregisterLocalFile removes the given filepath from the whitelist.
// DeregisterLocalFile removes the given filepath from the allowlist.
func DeregisterLocalFile(filePath string) {
fileRegisterLock.Lock()
delete(fileRegister, strings.Trim(filePath, `"`))

View File

@@ -28,11 +28,11 @@ func (nt *NullTime) Scan(value interface{}) (err error) {
nt.Time, nt.Valid = v, true
return
case []byte:
nt.Time, err = parseDateTime(string(v), time.UTC)
nt.Time, err = parseDateTime(v, time.UTC)
nt.Valid = (err == nil)
return
case string:
nt.Time, err = parseDateTime(v, time.UTC)
nt.Time, err = parseDateTime([]byte(v), time.UTC)
nt.Valid = (err == nil)
return
}

View File

@@ -28,4 +28,13 @@ import (
// }
//
// This NullTime implementation is not driver-specific
//
// Deprecated: NullTime doesn't honor the loc DSN parameter.
// NullTime.Scan interprets a time as UTC, not the loc DSN parameter.
// Use sql.NullTime instead.
type NullTime sql.NullTime
// for internal use.
// the mysql package uses sql.NullTime if it is available.
// if not, the package uses mysql.NullTime.
type nullTime = sql.NullTime // sql.NullTime is available

View File

@@ -32,3 +32,8 @@ type NullTime struct {
Time time.Time
Valid bool // Valid is true if Time is not NULL
}
// for internal use.
// the mysql package uses sql.NullTime if it is available.
// if not, the package uses mysql.NullTime.
type nullTime = NullTime // sql.NullTime is not available

View File

@@ -13,6 +13,7 @@ import (
"crypto/tls"
"database/sql/driver"
"encoding/binary"
"encoding/json"
"errors"
"fmt"
"io"
@@ -348,6 +349,12 @@ func (mc *mysqlConn) writeHandshakeResponsePacket(authResp []byte, plugin string
return errors.New("unknown collation")
}
// Filler [23 bytes] (all 0x00)
pos := 13
for ; pos < 13+23; pos++ {
data[pos] = 0
}
// SSL Connection Request Packet
// http://dev.mysql.com/doc/internals/en/connection-phase-packets.html#packet-Protocol::SSLRequest
if mc.cfg.tls != nil {
@@ -366,12 +373,6 @@ func (mc *mysqlConn) writeHandshakeResponsePacket(authResp []byte, plugin string
mc.buf.nc = tlsConn
}
// Filler [23 bytes] (all 0x00)
pos := 13
for ; pos < 13+23; pos++ {
data[pos] = 0
}
// User [null terminated string]
if len(mc.cfg.User) > 0 {
pos += copy(data[pos:], mc.cfg.User)
@@ -777,7 +778,7 @@ func (rows *textRows) readRow(dest []driver.Value) error {
case fieldTypeTimestamp, fieldTypeDateTime,
fieldTypeDate, fieldTypeNewDate:
dest[i], err = parseDateTime(
string(dest[i].([]byte)),
dest[i].([]byte),
mc.cfg.Loc,
)
if err == nil {
@@ -1003,6 +1004,9 @@ func (stmt *mysqlStmt) writeExecutePacket(args []driver.Value) error {
continue
}
if v, ok := arg.(json.RawMessage); ok {
arg = []byte(v)
}
// cache types and values
switch v := arg.(type) {
case int64:
@@ -1112,7 +1116,10 @@ func (stmt *mysqlStmt) writeExecutePacket(args []driver.Value) error {
if v.IsZero() {
b = append(b, "0000-00-00"...)
} else {
b = v.In(mc.cfg.Loc).AppendFormat(b, timeFormat)
b, err = appendDateTime(b, v.In(mc.cfg.Loc))
if err != nil {
return err
}
}
paramValues = appendLengthEncodedInteger(paramValues,

View File

@@ -10,6 +10,7 @@ package mysql
import (
"database/sql/driver"
"encoding/json"
"fmt"
"io"
"reflect"
@@ -43,6 +44,11 @@ func (stmt *mysqlStmt) ColumnConverter(idx int) driver.ValueConverter {
return converter{}
}
func (stmt *mysqlStmt) CheckNamedValue(nv *driver.NamedValue) (err error) {
nv.Value, err = converter{}.ConvertValue(nv.Value)
return
}
func (stmt *mysqlStmt) Exec(args []driver.Value) (driver.Result, error) {
if stmt.mc.closed.IsSet() {
errLog.Print(ErrInvalidConn)
@@ -129,6 +135,8 @@ func (stmt *mysqlStmt) query(args []driver.Value) (*binaryRows, error) {
return rows, err
}
var jsonType = reflect.TypeOf(json.RawMessage{})
type converter struct{}
// ConvertValue mirrors the reference/default converter in database/sql/driver
@@ -146,12 +154,17 @@ func (c converter) ConvertValue(v interface{}) (driver.Value, error) {
if err != nil {
return nil, err
}
if !driver.IsValue(sv) {
return nil, fmt.Errorf("non-Value type %T returned from Value", sv)
if driver.IsValue(sv) {
return sv, nil
}
return sv, nil
// A value returend from the Valuer interface can be "a type handled by
// a database driver's NamedValueChecker interface" so we should accept
// uint64 here as well.
if u, ok := sv.(uint64); ok {
return u, nil
}
return nil, fmt.Errorf("non-Value type %T returned from Value", sv)
}
rv := reflect.ValueOf(v)
switch rv.Kind() {
case reflect.Ptr:
@@ -170,11 +183,14 @@ func (c converter) ConvertValue(v interface{}) (driver.Value, error) {
case reflect.Bool:
return rv.Bool(), nil
case reflect.Slice:
ek := rv.Type().Elem().Kind()
if ek == reflect.Uint8 {
switch t := rv.Type(); {
case t == jsonType:
return v, nil
case t.Elem().Kind() == reflect.Uint8:
return rv.Bytes(), nil
default:
return nil, fmt.Errorf("unsupported type %T, a slice of %s", v, t.Elem().Kind())
}
return nil, fmt.Errorf("unsupported type %T, a slice of %s", v, ek)
case reflect.String:
return rv.String(), nil
}

View File

@@ -106,27 +106,136 @@ func readBool(input string) (value bool, valid bool) {
* Time related utils *
******************************************************************************/
func parseDateTime(str string, loc *time.Location) (t time.Time, err error) {
base := "0000-00-00 00:00:00.0000000"
switch len(str) {
func parseDateTime(b []byte, loc *time.Location) (time.Time, error) {
const base = "0000-00-00 00:00:00.000000"
switch len(b) {
case 10, 19, 21, 22, 23, 24, 25, 26: // up to "YYYY-MM-DD HH:MM:SS.MMMMMM"
if str == base[:len(str)] {
return
if string(b) == base[:len(b)] {
return time.Time{}, nil
}
t, err = time.Parse(timeFormat[:len(str)], str)
year, err := parseByteYear(b)
if err != nil {
return time.Time{}, err
}
if year <= 0 {
year = 1
}
if b[4] != '-' {
return time.Time{}, fmt.Errorf("bad value for field: `%c`", b[4])
}
m, err := parseByte2Digits(b[5], b[6])
if err != nil {
return time.Time{}, err
}
if m <= 0 {
m = 1
}
month := time.Month(m)
if b[7] != '-' {
return time.Time{}, fmt.Errorf("bad value for field: `%c`", b[7])
}
day, err := parseByte2Digits(b[8], b[9])
if err != nil {
return time.Time{}, err
}
if day <= 0 {
day = 1
}
if len(b) == 10 {
return time.Date(year, month, day, 0, 0, 0, 0, loc), nil
}
if b[10] != ' ' {
return time.Time{}, fmt.Errorf("bad value for field: `%c`", b[10])
}
hour, err := parseByte2Digits(b[11], b[12])
if err != nil {
return time.Time{}, err
}
if b[13] != ':' {
return time.Time{}, fmt.Errorf("bad value for field: `%c`", b[13])
}
min, err := parseByte2Digits(b[14], b[15])
if err != nil {
return time.Time{}, err
}
if b[16] != ':' {
return time.Time{}, fmt.Errorf("bad value for field: `%c`", b[16])
}
sec, err := parseByte2Digits(b[17], b[18])
if err != nil {
return time.Time{}, err
}
if len(b) == 19 {
return time.Date(year, month, day, hour, min, sec, 0, loc), nil
}
if b[19] != '.' {
return time.Time{}, fmt.Errorf("bad value for field: `%c`", b[19])
}
nsec, err := parseByteNanoSec(b[20:])
if err != nil {
return time.Time{}, err
}
return time.Date(year, month, day, hour, min, sec, nsec, loc), nil
default:
err = fmt.Errorf("invalid time string: %s", str)
return
return time.Time{}, fmt.Errorf("invalid time bytes: %s", b)
}
}
// Adjust location
if err == nil && loc != time.UTC {
y, mo, d := t.Date()
h, mi, s := t.Clock()
t, err = time.Date(y, mo, d, h, mi, s, t.Nanosecond(), loc), nil
func parseByteYear(b []byte) (int, error) {
year, n := 0, 1000
for i := 0; i < 4; i++ {
v, err := bToi(b[i])
if err != nil {
return 0, err
}
year += v * n
n = n / 10
}
return year, nil
}
return
func parseByte2Digits(b1, b2 byte) (int, error) {
d1, err := bToi(b1)
if err != nil {
return 0, err
}
d2, err := bToi(b2)
if err != nil {
return 0, err
}
return d1*10 + d2, nil
}
func parseByteNanoSec(b []byte) (int, error) {
ns, digit := 0, 100000 // max is 6-digits
for i := 0; i < len(b); i++ {
v, err := bToi(b[i])
if err != nil {
return 0, err
}
ns += v * digit
digit /= 10
}
// nanoseconds has 10-digits. (needs to scale digits)
// 10 - 6 = 4, so we have to multiple 1000.
return ns * 1000, nil
}
func bToi(b byte) (int, error) {
if b < '0' || b > '9' {
return 0, errors.New("not [0-9]")
}
return int(b - '0'), nil
}
func parseBinaryDateTime(num uint64, data []byte, loc *time.Location) (driver.Value, error) {
@@ -167,6 +276,64 @@ func parseBinaryDateTime(num uint64, data []byte, loc *time.Location) (driver.Va
return nil, fmt.Errorf("invalid DATETIME packet length %d", num)
}
func appendDateTime(buf []byte, t time.Time) ([]byte, error) {
year, month, day := t.Date()
hour, min, sec := t.Clock()
nsec := t.Nanosecond()
if year < 1 || year > 9999 {
return buf, errors.New("year is not in the range [1, 9999]: " + strconv.Itoa(year)) // use errors.New instead of fmt.Errorf to avoid year escape to heap
}
year100 := year / 100
year1 := year % 100
var localBuf [len("2006-01-02T15:04:05.999999999")]byte // does not escape
localBuf[0], localBuf[1], localBuf[2], localBuf[3] = digits10[year100], digits01[year100], digits10[year1], digits01[year1]
localBuf[4] = '-'
localBuf[5], localBuf[6] = digits10[month], digits01[month]
localBuf[7] = '-'
localBuf[8], localBuf[9] = digits10[day], digits01[day]
if hour == 0 && min == 0 && sec == 0 && nsec == 0 {
return append(buf, localBuf[:10]...), nil
}
localBuf[10] = ' '
localBuf[11], localBuf[12] = digits10[hour], digits01[hour]
localBuf[13] = ':'
localBuf[14], localBuf[15] = digits10[min], digits01[min]
localBuf[16] = ':'
localBuf[17], localBuf[18] = digits10[sec], digits01[sec]
if nsec == 0 {
return append(buf, localBuf[:19]...), nil
}
nsec100000000 := nsec / 100000000
nsec1000000 := (nsec / 1000000) % 100
nsec10000 := (nsec / 10000) % 100
nsec100 := (nsec / 100) % 100
nsec1 := nsec % 100
localBuf[19] = '.'
// milli second
localBuf[20], localBuf[21], localBuf[22] =
digits01[nsec100000000], digits10[nsec1000000], digits01[nsec1000000]
// micro second
localBuf[23], localBuf[24], localBuf[25] =
digits10[nsec10000], digits01[nsec10000], digits10[nsec100]
// nano second
localBuf[26], localBuf[27], localBuf[28] =
digits01[nsec100], digits10[nsec1], digits01[nsec1]
// trim trailing zeros
n := len(localBuf)
for n > 0 && localBuf[n-1] == '0' {
n--
}
return append(buf, localBuf[:n]...), nil
}
// zeroDateTime is used in formatBinaryDateTime to avoid an allocation
// if the DATE or DATETIME has the zero value.
// It must never be changed.

32
vendor/github.com/goccy/go-json/.codecov.yml generated vendored Normal file
View File

@@ -0,0 +1,32 @@
codecov:
require_ci_to_pass: yes
coverage:
precision: 2
round: down
range: "70...100"
status:
project:
default:
target: 70%
threshold: 2%
patch: off
changes: no
parsers:
gcov:
branch_detection:
conditional: yes
loop: yes
method: no
macro: no
comment:
layout: "header,diff"
behavior: default
require_changes: no
ignore:
- internal/encoder/vm_color
- internal/encoder/vm_color_indent

2
vendor/github.com/goccy/go-json/.gitignore generated vendored Normal file
View File

@@ -0,0 +1,2 @@
cover.html
cover.out

75
vendor/github.com/goccy/go-json/.golangci.yml generated vendored Normal file
View File

@@ -0,0 +1,75 @@
run:
skip-files:
- encode_optype.go
- ".*_test\\.go$"
linters-settings:
govet:
enable-all: true
disable:
- shadow
linters:
enable-all: true
disable:
- dogsled
- dupl
- exhaustive
- exhaustivestruct
- errorlint
- forbidigo
- funlen
- gci
- gochecknoglobals
- gochecknoinits
- gocognit
- gocritic
- gocyclo
- godot
- godox
- goerr113
- gofumpt
- gomnd
- gosec
- ifshort
- lll
- makezero
- nakedret
- nestif
- nlreturn
- paralleltest
- testpackage
- thelper
- wrapcheck
- interfacer
- lll
- nakedret
- nestif
- nlreturn
- testpackage
- wsl
issues:
exclude-rules:
# not needed
- path: /*.go
text: "ST1003: should not use underscores in package names"
linters:
- stylecheck
- path: /*.go
text: "don't use an underscore in package name"
linters:
- golint
- path: rtype.go
linters:
- golint
- stylecheck
- path: error.go
linters:
- staticcheck
# Maximum issues count per one linter. Set to 0 to disable. Default is 50.
max-issues-per-linter: 0
# Maximum count of issues with the same text. Set to 0 to disable. Default is 3.
max-same-issues: 0

236
vendor/github.com/goccy/go-json/CHANGELOG.md generated vendored Normal file
View File

@@ -0,0 +1,236 @@
# v0.7.4 - 2021/07/06
* Fix encoding of indirect layout structure ( #264 )
# v0.7.3 - 2021/06/29
* Fix encoding of pointer type in empty interface ( #262 )
# v0.7.2 - 2021/06/26
### Fix decoder
* Add decoder for func type to fix decoding of nil function value ( #257 )
* Fix stream decoding of []byte type ( #258 )
### Performance
* Improve decoding performance of map[string]interface{} type ( use `mapassign_faststr` ) ( #256 )
* Improve encoding performance of empty interface type ( remove recursive calling of `vm.Run` ) ( #259 )
### Benchmark
* Add bytedance/sonic as benchmark target ( #254 )
# v0.7.1 - 2021/06/18
### Fix decoder
* Fix error when unmarshal empty array ( #253 )
# v0.7.0 - 2021/06/12
### Support context for MarshalJSON and UnmarshalJSON ( #248 )
* json.MarshalContext(context.Context, interface{}, ...json.EncodeOption) ([]byte, error)
* json.NewEncoder(io.Writer).EncodeContext(context.Context, interface{}, ...json.EncodeOption) error
* json.UnmarshalContext(context.Context, []byte, interface{}, ...json.DecodeOption) error
* json.NewDecoder(io.Reader).DecodeContext(context.Context, interface{}) error
```go
type MarshalerContext interface {
MarshalJSON(context.Context) ([]byte, error)
}
type UnmarshalerContext interface {
UnmarshalJSON(context.Context, []byte) error
}
```
### Add DecodeFieldPriorityFirstWin option ( #242 )
In the default behavior, go-json, like encoding/json, will reflect the result of the last evaluation when a field with the same name exists. I've added new options to allow you to change this behavior. `json.DecodeFieldPriorityFirstWin` option reflects the result of the first evaluation if a field with the same name exists. This behavior has a performance advantage as it allows the subsequent strings to be skipped if all fields have been evaluated.
### Fix encoder
* Fix indent number contains recursive type ( #249 )
* Fix encoding of using empty interface as map key ( #244 )
### Fix decoder
* Fix decoding fields containing escaped characters ( #237 )
### Refactor
* Move some tests to subdirectory ( #243 )
* Refactor package layout for decoder ( #238 )
# v0.6.1 - 2021/06/02
### Fix encoder
* Fix value of totalLength for encoding ( #236 )
# v0.6.0 - 2021/06/01
### Support Colorize option for encoding (#233)
```go
b, err := json.MarshalWithOption(v, json.Colorize(json.DefaultColorScheme))
if err != nil {
...
}
fmt.Println(string(b)) // print colored json
```
### Refactor
* Fix opcode layout - Adjust memory layout of the opcode to 128 bytes in a 64-bit environment ( #230 )
* Refactor encode option ( #231 )
* Refactor escape string ( #232 )
# v0.5.1 - 2021/5/20
### Optimization
* Add type addrShift to enable bigger encoder/decoder cache ( #213 )
### Fix decoder
* Keep original reference of slice element ( #229 )
### Refactor
* Refactor Debug mode for encoding ( #226 )
* Generate VM sources for encoding ( #227 )
* Refactor validator for null/true/false for decoding ( #221 )
# v0.5.0 - 2021/5/9
### Supports using omitempty and string tags at the same time ( #216 )
### Fix decoder
* Fix stream decoder for unicode char ( #215 )
* Fix decoding of slice element ( #219 )
* Fix calculating of buffer length for stream decoder ( #220 )
### Refactor
* replace skipWhiteSpace goto by loop ( #212 )
# v0.4.14 - 2021/5/4
### Benchmark
* Add valyala/fastjson to benchmark ( #193 )
* Add benchmark task for CI ( #211 )
### Fix decoder
* Fix decoding of slice with unmarshal json type ( #198 )
* Fix decoding of null value for interface type that does not implement Unmarshaler ( #205 )
* Fix decoding of null value to []byte by json.Unmarshal ( #206 )
* Fix decoding of backslash char at the end of string ( #207 )
* Fix stream decoder for null/true/false value ( #208 )
* Fix stream decoder for slow reader ( #211 )
### Performance
* If cap of slice is enough, reuse slice data for compatibility with encoding/json ( #200 )
# v0.4.13 - 2021/4/20
### Fix json.Compact and json.Indent
* Support validation the input buffer for json.Compact and json.Indent ( #189 )
* Optimize json.Compact and json.Indent ( improve memory footprint ) ( #190 )
# v0.4.12 - 2021/4/15
### Fix encoder
* Fix unnecessary indent for empty slice type ( #181 )
* Fix encoding of omitempty feature for the slice or interface type ( #183 )
* Fix encoding custom types zero values with omitempty when marshaller exists ( #187 )
### Fix decoder
* Fix decoder for invalid top level value ( #184 )
* Fix decoder for invalid number value ( #185 )
# v0.4.11 - 2021/4/3
* Improve decoder performance for interface type
# v0.4.10 - 2021/4/2
### Fix encoder
* Fixed a bug when encoding slice and map containing recursive structures
* Fixed a logic to determine if indirect reference
# v0.4.9 - 2021/3/29
### Add debug mode
If you use `json.MarshalWithOption(v, json.Debug())` and `panic` occurred in `go-json`, produces debug information to console.
### Support a new feature to compatible with encoding/json
- invalid UTF-8 is coerced to valid UTF-8 ( without performance down )
### Fix encoder
- Fixed handling of MarshalJSON of function type
### Fix decoding of slice of pointer type
If there is a pointer value, go-json will use it. (This behavior is necessary to achieve the ability to prioritize pre-filled values). However, since slices are reused internally, there was a bug that referred to the previous pointer value. Therefore, it is not necessary to refer to the pointer value in advance for the slice element, so we explicitly initialize slice element by `nil`.
# v0.4.8 - 2021/3/21
### Reduce memory usage at compile time
* go-json have used about 2GB of memory at compile time, but now it can compile with about less than 550MB.
### Fix any encoder's bug
* Add many test cases for encoder
* Fix composite type ( slice/array/map )
* Fix pointer types
* Fix encoding of MarshalJSON or MarshalText or json.Number type
### Refactor encoder
* Change package layout for reducing memory usage at compile
* Remove anonymous and only operation
* Remove root property from encodeCompileContext and opcode
### Fix CI
* Add Go 1.16
* Remove Go 1.13
* Fix `make cover` task
### Number/Delim/Token/RawMessage use the types defined in encoding/json by type alias
# v0.4.7 - 2021/02/22
### Fix decoder
* Fix decoding of deep recursive structure
* Fix decoding of embedded unexported pointer field
* Fix invalid test case
* Fix decoding of invalid value
* Fix decoding of prefilled value
* Fix not being able to return UnmarshalTypeError when it should be returned
* Fix decoding of null value
* Fix decoding of type of null string
* Use pre allocated pointer if exists it at decoding
### Reduce memory usage at compile
* Integrate int/int8/int16/int32/int64 and uint/uint8/uint16/uint32/uint64 operation to reduce memory usage at compile
### Remove unnecessary optype

21
vendor/github.com/goccy/go-json/LICENSE generated vendored Normal file
View File

@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2020 Masaaki Goshima
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

39
vendor/github.com/goccy/go-json/Makefile generated vendored Normal file
View File

@@ -0,0 +1,39 @@
PKG := github.com/goccy/go-json
BIN_DIR := $(CURDIR)/bin
PKGS := $(shell go list ./... | grep -v internal/cmd|grep -v test)
COVER_PKGS := $(foreach pkg,$(PKGS),$(subst $(PKG),.,$(pkg)))
COMMA := ,
EMPTY :=
SPACE := $(EMPTY) $(EMPTY)
COVERPKG_OPT := $(subst $(SPACE),$(COMMA),$(COVER_PKGS))
$(BIN_DIR):
@mkdir -p $(BIN_DIR)
.PHONY: cover
cover:
go test -coverpkg=$(COVERPKG_OPT) -coverprofile=cover.out ./...
.PHONY: cover-html
cover-html: cover
go tool cover -html=cover.out
.PHONY: lint
lint: golangci-lint
golangci-lint run
golangci-lint: | $(BIN_DIR)
@{ \
set -e; \
GOLANGCI_LINT_TMP_DIR=$$(mktemp -d); \
cd $$GOLANGCI_LINT_TMP_DIR; \
go mod init tmp; \
GOBIN=$(BIN_DIR) go get github.com/golangci/golangci-lint/cmd/golangci-lint@v1.36.0; \
rm -rf $$GOLANGCI_LINT_TMP_DIR; \
}
.PHONY: generate
generate:
go generate ./internal/...

529
vendor/github.com/goccy/go-json/README.md generated vendored Normal file
View File

@@ -0,0 +1,529 @@
# go-json
![Go](https://github.com/goccy/go-json/workflows/Go/badge.svg)
[![GoDoc](https://godoc.org/github.com/goccy/go-json?status.svg)](https://pkg.go.dev/github.com/goccy/go-json?tab=doc)
[![codecov](https://codecov.io/gh/goccy/go-json/branch/master/graph/badge.svg)](https://codecov.io/gh/goccy/go-json)
Fast JSON encoder/decoder compatible with encoding/json for Go
<img width="400px" src="https://user-images.githubusercontent.com/209884/92572337-42b42900-f2bf-11ea-973a-c74a359553a5.png"></img>
# Roadmap
```
* version ( expected release date )
* v0.7.0
|
| while maintaining compatibility with encoding/json, we will add convenient APIs
|
v
* v1.0.0
```
We are accepting requests for features that will be implemented between v0.7.0 and v.1.0.0.
If you have the API you need, please submit your issue [here](https://github.com/goccy/go-json/issues).
For example, I'm thinking of supporting `context.Context` of `json.Marshaler` and decoding using JSON Path.
# Features
- Drop-in replacement of `encoding/json`
- Fast ( See [Benchmark section](https://github.com/goccy/go-json#benchmarks) )
- Flexible customization with options
- Coloring the encoded string
- Can propagate context.Context to `MarshalJSON` or `UnmarshalJSON`
# Installation
```
go get github.com/goccy/go-json
```
# How to use
Replace import statement from `encoding/json` to `github.com/goccy/go-json`
```
-import "encoding/json"
+import "github.com/goccy/go-json"
```
# JSON library comparison
| name | encoder | decoder | compatible with `encoding/json` |
| :----: | :------: | :-----: | :-----------------------------: |
| encoding/json | yes | yes | N/A |
| [json-iterator/go](https://github.com/json-iterator/go) | yes | yes | partial |
| [easyjson](https://github.com/mailru/easyjson) | yes | yes | no |
| [gojay](https://github.com/francoispqt/gojay) | yes | yes | no |
| [segmentio/encoding/json](https://github.com/segmentio/encoding/tree/master/json) | yes | yes | partial |
| [jettison](https://github.com/wI2L/jettison) | yes | no | no |
| [simdjson-go](https://github.com/minio/simdjson-go) | no | yes | no |
| goccy/go-json | yes | yes | yes |
- `json-iterator/go` isn't compatible with `encoding/json` in many ways (e.g. https://github.com/json-iterator/go/issues/229 ), but it hasn't been supported for a long time.
- `segmentio/encoding/json` is well supported for encoders, but some are not supported for decoder APIs such as `Token` ( streaming decode )
## Other libraries
- [jingo](https://github.com/bet365/jingo)
I tried the benchmark but it didn't work.
Also, it seems to panic when it receives an unexpected value because there is no error handling...
- [ffjson](https://github.com/pquerna/ffjson)
Benchmarking gave very slow results.
It seems that it is assumed that the user will use the buffer pool properly.
Also, development seems to have already stopped
# Benchmarks
```
$ cd benchmarks
$ go test -bench .
```
## Encode
<img width="700px" src="https://user-images.githubusercontent.com/209884/107126758-0845cb00-68f5-11eb-8db7-086fcf9bcfaa.png"></img>
<img width="700px" src="https://user-images.githubusercontent.com/209884/107126757-07ad3480-68f5-11eb-87aa-858cc5eacfcb.png"></img>
## Decode
<img width="700" alt="" src="https://user-images.githubusercontent.com/209884/107979944-bd1d6d80-7002-11eb-944b-9d17b6674e3f.png">
<img width="700" alt="" src="https://user-images.githubusercontent.com/209884/107979931-b989e680-7002-11eb-87a0-66fc22d90dd4.png">
<img width="700" alt="" src="https://user-images.githubusercontent.com/209884/107979940-bc84d700-7002-11eb-9647-869bbc25c9d9.png">
# Fuzzing
[go-json-fuzz](https://github.com/goccy/go-json-fuzz) is the repository for fuzzing tests.
If you run the test in this repository and find a bug, please commit to corpus to go-json-fuzz and report the issue to [go-json](https://github.com/goccy/go-json/issues).
# How it works
`go-json` is very fast in both encoding and decoding compared to other libraries.
It's easier to implement by using automatic code generation for performance or by using a dedicated interface, but `go-json` dares to stick to compatibility with `encoding/json` and is the simple interface. Despite this, we are developing with the aim of being the fastest library.
Here, we explain the various speed-up techniques implemented by `go-json`.
## Basic technique
The techniques listed here are the ones used by most of the libraries listed above.
### Buffer reuse
Since the only value required for the result of `json.Marshal(interface{}) ([]byte, error)` is `[]byte`, the only value that must be allocated during encoding is the return value `[]byte` .
Also, as the number of allocations increases, the performance will be affected, so the number of allocations should be kept as low as possible when creating `[]byte`.
Therefore, there is a technique to reduce the number of times a new buffer must be allocated by reusing the buffer used for the previous encoding by using `sync.Pool`.
Finally, you allocate a buffer that is as long as the resulting buffer and copy the contents into it, you only need to allocate the buffer once in theory.
```go
type buffer struct {
data []byte
}
var bufPool = sync.Pool{
New: func() interface{} {
return &buffer{data: make([]byte, 0, 1024)}
},
}
buf := bufPool.Get().(*buffer)
data := encode(buf.data) // reuse buf.data
newBuf := make([]byte, len(data))
copy(newBuf, buf)
buf.data = data
bufPool.Put(buf)
```
### Elimination of reflection
As you know, the reflection operation is very slow.
Therefore, using the fact that the address position where the type information is stored is fixed for each binary ( we call this `typeptr` ),
we can use the address in the type information to call a pre-built optimized process.
For example, you can get the address to the type information from `interface{}` as follows and you can use that information to call a process that does not have reflection.
To process without reflection, pass a pointer (`unsafe.Pointer`) to the value is stored.
```go
type emptyInterface struct {
typ unsafe.Pointer
ptr unsafe.Pointer
}
var typeToEncoder = map[uintptr]func(unsafe.Pointer)([]byte, error){}
func Marshal(v interface{}) ([]byte, error) {
iface := (*emptyInterface)(unsafe.Pointer(&v)
typeptr := uintptr(iface.typ)
if enc, exists := typeToEncoder[typeptr]; exists {
return enc(iface.ptr)
}
...
}
```
※ In reality, `typeToEncoder` can be referenced by multiple goroutines, so exclusive control is required.
## Unique speed-up technique
## Encoder
### Do not escape arguments of `Marshal`
`json.Marshal` and `json.Unmarshal` receive `interface{}` value and they perform type determination dynamically to process.
In normal case, you need to use the `reflect` library to determine the type dynamically, but since `reflect.Type` is defined as `interface`, when you call the method of `reflect.Type`, The reflect's argument is escaped.
Therefore, the arguments for `Marshal` and `Unmarshal` are always escape to the heap.
However, `go-json` can use the feature of `reflect.Type` while avoiding escaping.
`reflect.Type` is defined as `interface`, but in reality `reflect.Type` is implemented only by the structure `rtype` defined in the `reflect` package.
For this reason, to date `reflect.Type` is the same as `*reflect.rtype`.
Therefore, by directly handling `*reflect.rtype`, which is an implementation of `reflect.Type`, it is possible to avoid escaping because it changes from `interface` to using `struct`.
The technique for working with `*reflect.rtype` directly from `go-json` is implemented at [rtype.go](https://github.com/goccy/go-json/blob/master/internal/runtime/rtype.go)
Also, the same technique is cut out as a library ( https://github.com/goccy/go-reflect )
Initially this feature was the default behavior of `go-json`.
But after careful testing, I found that I passed a large value to `json.Marshal()` and if the argument could not be assigned to the stack, it could not be properly escaped to the heap (a bug in the Go compiler).
Therefore, this feature will be provided as an **optional** until this issue is resolved.
To use it, add `NoEscape` like `MarshalNoEscape()`
### Encoding using opcode sequence
I explained that you can use `typeptr` to call a pre-built process from type information.
In other libraries, this dedicated process is processed by making it an function calling like anonymous function, but function calls are inherently slow processes and should be avoided as much as possible.
Therefore, `go-json` adopted the Instruction-based execution processing system, which is also used to implement virtual machines for programming language.
If it is the first type to encode, create the opcode ( instruction ) sequence required for encoding.
From the second time onward, use `typeptr` to get the cached pre-built opcode sequence and encode it based on it. An example of the opcode sequence is shown below.
```go
json.Marshal(struct{
X int `json:"x"`
Y string `json:"y"`
}{X: 1, Y: "hello"})
```
When encoding a structure like the one above, create a sequence of opcodes like this:
```
- opStructFieldHead ( `{` )
- opStructFieldInt ( `"x": 1,` )
- opStructFieldString ( `"y": "hello"` )
- opStructEnd ( `}` )
- opEnd
```
※ When processing each operation, write the letters on the right.
In addition, each opcode is managed by the following structure (
Pseudo code ).
```go
type opType int
const (
opStructFieldHead opType = iota
opStructFieldInt
opStructFieldStirng
opStructEnd
opEnd
)
type opcode struct {
op opType
key []byte
next *opcode
}
```
The process of encoding using the opcode sequence is roughly implemented as follows.
```go
func encode(code *opcode, b []byte, p unsafe.Pointer) ([]byte, error) {
for {
switch code.op {
case opStructFieldHead:
b = append(b, '{')
code = code.next
case opStructFieldInt:
b = append(b, code.key...)
b = appendInt((*int)(unsafe.Pointer(uintptr(p)+code.offset)))
code = code.next
case opStructFieldString:
b = append(b, code.key...)
b = appendString((*string)(unsafe.Pointer(uintptr(p)+code.offset)))
code = code.next
case opStructEnd:
b = append(b, '}')
code = code.next
case opEnd:
goto END
}
}
END:
return b, nil
}
```
In this way, the huge `switch-case` is used to encode by manipulating the linked list opcodes to avoid unnecessary function calls.
### Opcode sequence optimization
One of the advantages of encoding using the opcode sequence is the ease of optimization.
The opcode sequence mentioned above is actually converted into the following optimized operations and used.
```
- opStructFieldHeadInt ( `{"x": 1,` )
- opStructEndString ( `"y": "hello"}` )
- opEnd
```
It has been reduced from 5 opcodes to 3 opcodes !
Reducing the number of opcodees means reducing the number of branches with `switch-case`.
In other words, the closer the number of operations is to 1, the faster the processing can be performed.
In `go-json`, optimization to reduce the number of opcodes itself like the above and it speeds up by preparing opcodes with optimized paths.
### Change recursive call from CALL to JMP
Recursive processing is required during encoding if the type is defined recursively as follows:
```go
type T struct {
X int
U *U
}
type U struct {
T *T
}
b, err := json.Marshal(&T{
X: 1,
U: &U{
T: &T{
X: 2,
},
},
})
fmt.Println(string(b)) // {"X":1,"U":{"T":{"X":2,"U":null}}}
```
In `go-json`, recursive processing is processed by the operation type of ` opStructFieldRecursive`.
In this operation, after acquiring the opcode sequence used for recursive processing, the function is **not** called recursively as it is, but the necessary values are saved by itself and implemented by moving to the next operation.
The technique of implementing recursive processing with the `JMP` operation while avoiding the `CALL` operation is a famous technique for implementing a high-speed virtual machine.
For more details, please refer to [the article](https://engineering.mercari.com/blog/entry/1599563768-081104c850) ( but Japanese only ).
### Dispatch by typeptr from map to slice
When retrieving the data cached from the type information by `typeptr`, we usually use map.
Map requires exclusive control, so use `sync.Map` for a naive implementation.
However, this is slow, so it's a good idea to use the `atomic` package for exclusive control as implemented by `segmentio/encoding/json` ( https://github.com/segmentio/encoding/blob/master/json/codec.go#L41-L55 ).
This implementation slows down the set instead of speeding up the get, but it works well because of the nature of the library, it encodes much more for the same type.
However, as a result of profiling, I noticed that `runtime.mapaccess2` accounts for a significant percentage of the execution time. So I thought if I could change the lookup from map to slice.
There is an API named `typelinks` defined in the `runtime` package that the `reflect` package uses internally.
This allows you to get all the type information defined in the binary at runtime.
The fact that all type information can be acquired means that by constructing slices in advance with the acquired total number of type information, it is possible to look up with the value of `typeptr` without worrying about out-of-range access.
However, if there is too much type information, it will use a lot of memory, so by default we will only use this optimization if the slice size fits within **2Mib** .
If this approach is not available, it will fall back to the `atomic` based process described above.
If you want to know more, please refer to the implementation [here](https://github.com/goccy/go-json/blob/master/internal/runtime/type.go#L36-L100)
## Decoder
### Dispatch by typeptr from map to slice
Like the encoder, the decoder also uses typeptr to call the dedicated process.
### Faster termination character inspection using NUL character
In order to decode, you have to traverse the input buffer character by position.
At that time, if you check whether the buffer has reached the end, it will be very slow.
`buf` : `[]byte` type variable. holds the string passed to the decoder
`cursor` : `int64` type variable. holds the current read position
```go
buflen := len(buf)
for ; cursor < buflen; cursor++ { // compare cursor and buflen at all times, it is so slow.
switch buf[cursor] {
case ' ', '\n', '\r', '\t':
}
}
```
Therefore, by adding the `NUL` (`\000`) character to the end of the read buffer as shown below, it is possible to check the termination character at the same time as other characters.
```go
for {
switch buf[cursor] {
case ' ', '\n', '\r', '\t':
case '\000':
return nil
}
cursor++
}
```
### Use Boundary Check Elimination
Due to the `NUL` character optimization, the Go compiler does a boundary check every time, even though `buf[cursor]` does not cause out-of-range access.
Therefore, `go-json` eliminates boundary check by fetching characters for hotspot by pointer operation. For example, the following code.
```go
func char(ptr unsafe.Pointer, offset int64) byte {
return *(*byte)(unsafe.Pointer(uintptr(ptr) + uintptr(offset)))
}
p := (*sliceHeader)(&unsafe.Pointer(buf)).data
for {
switch char(p, cursor) {
case ' ', '\n', '\r', '\t':
case '\000':
return nil
}
cursor++
}
```
### Checking the existence of fields of struct using Bitmaps
I found by the profiling result, in the struct decode, lookup process for field was taking a long time.
For example, consider decoding a string like `{"a":1,"b":2,"c":3}` into the following structure:
```go
type T struct {
A int `json:"a"`
B int `json:"b"`
C int `json:"c"`
}
```
At this time, it was found that it takes a lot of time to acquire the decoding process corresponding to the field from the field name as shown below during the decoding process.
```go
fieldName := decodeKey(buf, cursor) // "a" or "b" or "c"
decoder, exists := fieldToDecoderMap[fieldName] // so slow
if exists {
decoder(buf, cursor)
} else {
skipValue(buf, cursor)
}
```
To improve this process, `json-iterator/go` is optimized so that it can be branched by switch-case when the number of fields in the structure is 10 or less (switch-case is faster than map). However, there is a risk of hash collision because the value hashed by the FNV algorithm is used for conditional branching. Also, `gojay` processes this part at high speed by letting the library user yourself write `switch-case`.
`go-json` considers and implements a new approach that is different from these. I call this **bitmap field optimization**.
The range of values per character can be represented by `[256]byte`. Also, if the number of fields in the structure is 8 or less, `int8` type can represent the state of each field.
In other words, it has the following structure.
- Base ( 8bit ): `00000000`
- Key "a": `00000001` ( assign key "a" to the first bit )
- Key "b": `00000010` ( assign key "b" to the second bit )
- Key "c": `00000100` ( assign key "c" to the third bit )
Bitmap structure is the following
```
| key index(0) |
------------------------
0 | 00000000 |
1 | 00000000 |
~~ | |
97 (a) | 00000001 |
98 (b) | 00000010 |
99 (c) | 00000100 |
~~ | |
255 | 00000000 |
```
You can think of this as a Bitmap with a height of `256` and a width of the maximum string length in the field name.
In other words, it can be represented by the following type .
```go
[maxFieldKeyLength][256]int8
```
When decoding a field character, check whether the corresponding character exists by referring to the pre-built bitmap like the following.
```go
var curBit int8 = math.MaxInt8 // 11111111
c := char(buf, cursor)
bit := bitmap[keyIdx][c]
curBit &= bit
if curBit == 0 {
// not found field
}
```
If `curBit` is not `0` until the end of the field string, then the string is
You may have hit one of the fields.
But the possibility is that if the decoded string is shorter than the field string, you will get a false hit.
- input: `{"a":1}`
```go
type T struct {
X int `json:"abc"`
}
```
※ Since `a` is shorter than `abc`, it can decode to the end of the field character without `curBit` being 0.
Rest assured. In this case, it doesn't matter because you can tell if you hit by comparing the string length of `a` with the string length of `abc`.
Finally, calculate the position of the bit where `1` is set and get the corresponding value, and you're done.
Using this technique, field lookups are possible with only bitwise operations and access to slices.
`go-json` uses a similar technique for fields with 9 or more and 16 or less fields. At this time, Bitmap is constructed as `[maxKeyLen][256]int16` type.
Currently, this optimization is not performed when the maximum length of the field name is long (specifically, 64 bytes or more) in addition to the limitation of the number of fields from the viewpoint of saving memory usage.
### Others
I have done a lot of other optimizations. I will find time to write about them. If you have any questions about what's written here or other optimizations, please visit the `#go-json` channel on `gophers.slack.com` .
## Reference
Regarding the story of go-json, there are the following articles in Japanese only.
- https://speakerdeck.com/goccy/zui-su-falsejsonraiburariwoqiu-mete
- https://engineering.mercari.com/blog/entry/1599563768-081104c850/
# Looking for Sponsors
I'm looking for sponsors this library. This library is being developed as a personal project in my spare time. If you want a quick response or problem resolution when using this library in your project, please register as a [sponsor](https://github.com/sponsors/goccy). I will cooperate as much as possible. Of course, this library is developed as an MIT license, so you can use it freely for free.
# License
MIT

68
vendor/github.com/goccy/go-json/color.go generated vendored Normal file
View File

@@ -0,0 +1,68 @@
package json
import (
"fmt"
"github.com/goccy/go-json/internal/encoder"
)
type (
ColorFormat = encoder.ColorFormat
ColorScheme = encoder.ColorScheme
)
const escape = "\x1b"
type colorAttr int
//nolint:deadcode,varcheck
const (
fgBlackColor colorAttr = iota + 30
fgRedColor
fgGreenColor
fgYellowColor
fgBlueColor
fgMagentaColor
fgCyanColor
fgWhiteColor
)
//nolint:deadcode,varcheck
const (
fgHiBlackColor colorAttr = iota + 90
fgHiRedColor
fgHiGreenColor
fgHiYellowColor
fgHiBlueColor
fgHiMagentaColor
fgHiCyanColor
fgHiWhiteColor
)
func createColorFormat(attr colorAttr) ColorFormat {
return ColorFormat{
Header: wrapColor(attr),
Footer: resetColor(),
}
}
func wrapColor(attr colorAttr) string {
return fmt.Sprintf("%s[%dm", escape, attr)
}
func resetColor() string {
return wrapColor(colorAttr(0))
}
var (
DefaultColorScheme = &ColorScheme{
Int: createColorFormat(fgHiMagentaColor),
Uint: createColorFormat(fgHiMagentaColor),
Float: createColorFormat(fgHiMagentaColor),
Bool: createColorFormat(fgHiYellowColor),
String: createColorFormat(fgHiGreenColor),
Binary: createColorFormat(fgHiRedColor),
ObjectKey: createColorFormat(fgHiCyanColor),
Null: createColorFormat(fgBlueColor),
}
)

232
vendor/github.com/goccy/go-json/decode.go generated vendored Normal file
View File

@@ -0,0 +1,232 @@
package json
import (
"context"
"fmt"
"io"
"reflect"
"unsafe"
"github.com/goccy/go-json/internal/decoder"
"github.com/goccy/go-json/internal/errors"
"github.com/goccy/go-json/internal/runtime"
)
type Decoder struct {
s *decoder.Stream
}
const (
nul = '\000'
)
type emptyInterface struct {
typ *runtime.Type
ptr unsafe.Pointer
}
func unmarshal(data []byte, v interface{}, optFuncs ...DecodeOptionFunc) error {
src := make([]byte, len(data)+1) // append nul byte to the end
copy(src, data)
header := (*emptyInterface)(unsafe.Pointer(&v))
if err := validateType(header.typ, uintptr(header.ptr)); err != nil {
return err
}
dec, err := decoder.CompileToGetDecoder(header.typ)
if err != nil {
return err
}
ctx := decoder.TakeRuntimeContext()
ctx.Buf = src
ctx.Option.Flags = 0
for _, optFunc := range optFuncs {
optFunc(ctx.Option)
}
cursor, err := dec.Decode(ctx, 0, 0, header.ptr)
if err != nil {
decoder.ReleaseRuntimeContext(ctx)
return err
}
decoder.ReleaseRuntimeContext(ctx)
return validateEndBuf(src, cursor)
}
func unmarshalContext(ctx context.Context, data []byte, v interface{}, optFuncs ...DecodeOptionFunc) error {
src := make([]byte, len(data)+1) // append nul byte to the end
copy(src, data)
header := (*emptyInterface)(unsafe.Pointer(&v))
if err := validateType(header.typ, uintptr(header.ptr)); err != nil {
return err
}
dec, err := decoder.CompileToGetDecoder(header.typ)
if err != nil {
return err
}
rctx := decoder.TakeRuntimeContext()
rctx.Buf = src
rctx.Option.Flags = 0
rctx.Option.Flags |= decoder.ContextOption
rctx.Option.Context = ctx
for _, optFunc := range optFuncs {
optFunc(rctx.Option)
}
cursor, err := dec.Decode(rctx, 0, 0, header.ptr)
if err != nil {
decoder.ReleaseRuntimeContext(rctx)
return err
}
decoder.ReleaseRuntimeContext(rctx)
return validateEndBuf(src, cursor)
}
func unmarshalNoEscape(data []byte, v interface{}, optFuncs ...DecodeOptionFunc) error {
src := make([]byte, len(data)+1) // append nul byte to the end
copy(src, data)
header := (*emptyInterface)(unsafe.Pointer(&v))
if err := validateType(header.typ, uintptr(header.ptr)); err != nil {
return err
}
dec, err := decoder.CompileToGetDecoder(header.typ)
if err != nil {
return err
}
ctx := decoder.TakeRuntimeContext()
ctx.Buf = src
ctx.Option.Flags = 0
for _, optFunc := range optFuncs {
optFunc(ctx.Option)
}
cursor, err := dec.Decode(ctx, 0, 0, noescape(header.ptr))
if err != nil {
decoder.ReleaseRuntimeContext(ctx)
return err
}
decoder.ReleaseRuntimeContext(ctx)
return validateEndBuf(src, cursor)
}
func validateEndBuf(src []byte, cursor int64) error {
for {
switch src[cursor] {
case ' ', '\t', '\n', '\r':
cursor++
continue
case nul:
return nil
}
return errors.ErrSyntax(
fmt.Sprintf("invalid character '%c' after top-level value", src[cursor]),
cursor+1,
)
}
}
//nolint:staticcheck
//go:nosplit
func noescape(p unsafe.Pointer) unsafe.Pointer {
x := uintptr(p)
return unsafe.Pointer(x ^ 0)
}
func validateType(typ *runtime.Type, p uintptr) error {
if typ == nil || typ.Kind() != reflect.Ptr || p == 0 {
return &InvalidUnmarshalError{Type: runtime.RType2Type(typ)}
}
return nil
}
// NewDecoder returns a new decoder that reads from r.
//
// The decoder introduces its own buffering and may
// read data from r beyond the JSON values requested.
func NewDecoder(r io.Reader) *Decoder {
s := decoder.NewStream(r)
return &Decoder{
s: s,
}
}
// Buffered returns a reader of the data remaining in the Decoder's
// buffer. The reader is valid until the next call to Decode.
func (d *Decoder) Buffered() io.Reader {
return d.s.Buffered()
}
// Decode reads the next JSON-encoded value from its
// input and stores it in the value pointed to by v.
//
// See the documentation for Unmarshal for details about
// the conversion of JSON into a Go value.
func (d *Decoder) Decode(v interface{}) error {
return d.DecodeWithOption(v)
}
// DecodeContext reads the next JSON-encoded value from its
// input and stores it in the value pointed to by v with context.Context.
func (d *Decoder) DecodeContext(ctx context.Context, v interface{}) error {
d.s.Option.Flags |= decoder.ContextOption
d.s.Option.Context = ctx
return d.DecodeWithOption(v)
}
func (d *Decoder) DecodeWithOption(v interface{}, optFuncs ...DecodeOptionFunc) error {
header := (*emptyInterface)(unsafe.Pointer(&v))
typ := header.typ
ptr := uintptr(header.ptr)
typeptr := uintptr(unsafe.Pointer(typ))
// noescape trick for header.typ ( reflect.*rtype )
copiedType := *(**runtime.Type)(unsafe.Pointer(&typeptr))
if err := validateType(copiedType, ptr); err != nil {
return err
}
dec, err := decoder.CompileToGetDecoder(typ)
if err != nil {
return err
}
if err := d.s.PrepareForDecode(); err != nil {
return err
}
s := d.s
for _, optFunc := range optFuncs {
optFunc(s.Option)
}
if err := dec.DecodeStream(s, 0, header.ptr); err != nil {
return err
}
s.Reset()
return nil
}
func (d *Decoder) More() bool {
return d.s.More()
}
func (d *Decoder) Token() (Token, error) {
return d.s.Token()
}
// DisallowUnknownFields causes the Decoder to return an error when the destination
// is a struct and the input contains object keys which do not match any
// non-ignored, exported fields in the destination.
func (d *Decoder) DisallowUnknownFields() {
d.s.DisallowUnknownFields = true
}
func (d *Decoder) InputOffset() int64 {
return d.s.TotalOffset()
}
// UseNumber causes the Decoder to unmarshal a number into an interface{} as a
// Number instead of as a float64.
func (d *Decoder) UseNumber() {
d.s.UseNumber = true
}

13
vendor/github.com/goccy/go-json/docker-compose.yml generated vendored Normal file
View File

@@ -0,0 +1,13 @@
version: '2'
services:
go-json:
image: golang:1.16
volumes:
- '.:/go/src/go-json'
deploy:
resources:
limits:
memory: 620M
working_dir: /go/src/go-json
command: |
sh -c "go test -c . && ls go-json.test"

323
vendor/github.com/goccy/go-json/encode.go generated vendored Normal file
View File

@@ -0,0 +1,323 @@
package json
import (
"context"
"io"
"unsafe"
"github.com/goccy/go-json/internal/encoder"
"github.com/goccy/go-json/internal/encoder/vm"
"github.com/goccy/go-json/internal/encoder/vm_color"
"github.com/goccy/go-json/internal/encoder/vm_color_indent"
"github.com/goccy/go-json/internal/encoder/vm_indent"
)
// An Encoder writes JSON values to an output stream.
type Encoder struct {
w io.Writer
enabledIndent bool
enabledHTMLEscape bool
prefix string
indentStr string
}
// NewEncoder returns a new encoder that writes to w.
func NewEncoder(w io.Writer) *Encoder {
return &Encoder{w: w, enabledHTMLEscape: true}
}
// Encode writes the JSON encoding of v to the stream, followed by a newline character.
//
// See the documentation for Marshal for details about the conversion of Go values to JSON.
func (e *Encoder) Encode(v interface{}) error {
return e.EncodeWithOption(v)
}
// EncodeWithOption call Encode with EncodeOption.
func (e *Encoder) EncodeWithOption(v interface{}, optFuncs ...EncodeOptionFunc) error {
ctx := encoder.TakeRuntimeContext()
ctx.Option.Flag = 0
err := e.encodeWithOption(ctx, v, optFuncs...)
encoder.ReleaseRuntimeContext(ctx)
return err
}
// EncodeContext call Encode with context.Context and EncodeOption.
func (e *Encoder) EncodeContext(ctx context.Context, v interface{}, optFuncs ...EncodeOptionFunc) error {
rctx := encoder.TakeRuntimeContext()
rctx.Option.Flag = 0
rctx.Option.Flag |= encoder.ContextOption
rctx.Option.Context = ctx
err := e.encodeWithOption(rctx, v, optFuncs...)
encoder.ReleaseRuntimeContext(rctx)
return err
}
func (e *Encoder) encodeWithOption(ctx *encoder.RuntimeContext, v interface{}, optFuncs ...EncodeOptionFunc) error {
if e.enabledHTMLEscape {
ctx.Option.Flag |= encoder.HTMLEscapeOption
}
for _, optFunc := range optFuncs {
optFunc(ctx.Option)
}
var (
buf []byte
err error
)
if e.enabledIndent {
buf, err = encodeIndent(ctx, v, e.prefix, e.indentStr)
} else {
buf, err = encode(ctx, v)
}
if err != nil {
return err
}
if e.enabledIndent {
buf = buf[:len(buf)-2]
} else {
buf = buf[:len(buf)-1]
}
buf = append(buf, '\n')
if _, err := e.w.Write(buf); err != nil {
return err
}
return nil
}
// SetEscapeHTML specifies whether problematic HTML characters should be escaped inside JSON quoted strings.
// The default behavior is to escape &, <, and > to \u0026, \u003c, and \u003e to avoid certain safety problems that can arise when embedding JSON in HTML.
//
// In non-HTML settings where the escaping interferes with the readability of the output, SetEscapeHTML(false) disables this behavior.
func (e *Encoder) SetEscapeHTML(on bool) {
e.enabledHTMLEscape = on
}
// SetIndent instructs the encoder to format each subsequent encoded value as if indented by the package-level function Indent(dst, src, prefix, indent).
// Calling SetIndent("", "") disables indentation.
func (e *Encoder) SetIndent(prefix, indent string) {
if prefix == "" && indent == "" {
e.enabledIndent = false
return
}
e.prefix = prefix
e.indentStr = indent
e.enabledIndent = true
}
func marshalContext(ctx context.Context, v interface{}, optFuncs ...EncodeOptionFunc) ([]byte, error) {
rctx := encoder.TakeRuntimeContext()
rctx.Option.Flag = 0
rctx.Option.Flag = encoder.HTMLEscapeOption | encoder.ContextOption
rctx.Option.Context = ctx
for _, optFunc := range optFuncs {
optFunc(rctx.Option)
}
buf, err := encode(rctx, v)
if err != nil {
encoder.ReleaseRuntimeContext(rctx)
return nil, err
}
// this line exists to escape call of `runtime.makeslicecopy` .
// if use `make([]byte, len(buf)-1)` and `copy(copied, buf)`,
// dst buffer size and src buffer size are differrent.
// in this case, compiler uses `runtime.makeslicecopy`, but it is slow.
buf = buf[:len(buf)-1]
copied := make([]byte, len(buf))
copy(copied, buf)
encoder.ReleaseRuntimeContext(rctx)
return copied, nil
}
func marshal(v interface{}, optFuncs ...EncodeOptionFunc) ([]byte, error) {
ctx := encoder.TakeRuntimeContext()
ctx.Option.Flag = 0
ctx.Option.Flag |= encoder.HTMLEscapeOption
for _, optFunc := range optFuncs {
optFunc(ctx.Option)
}
buf, err := encode(ctx, v)
if err != nil {
encoder.ReleaseRuntimeContext(ctx)
return nil, err
}
// this line exists to escape call of `runtime.makeslicecopy` .
// if use `make([]byte, len(buf)-1)` and `copy(copied, buf)`,
// dst buffer size and src buffer size are differrent.
// in this case, compiler uses `runtime.makeslicecopy`, but it is slow.
buf = buf[:len(buf)-1]
copied := make([]byte, len(buf))
copy(copied, buf)
encoder.ReleaseRuntimeContext(ctx)
return copied, nil
}
func marshalNoEscape(v interface{}) ([]byte, error) {
ctx := encoder.TakeRuntimeContext()
ctx.Option.Flag = 0
ctx.Option.Flag |= encoder.HTMLEscapeOption
buf, err := encodeNoEscape(ctx, v)
if err != nil {
encoder.ReleaseRuntimeContext(ctx)
return nil, err
}
// this line exists to escape call of `runtime.makeslicecopy` .
// if use `make([]byte, len(buf)-1)` and `copy(copied, buf)`,
// dst buffer size and src buffer size are differrent.
// in this case, compiler uses `runtime.makeslicecopy`, but it is slow.
buf = buf[:len(buf)-1]
copied := make([]byte, len(buf))
copy(copied, buf)
encoder.ReleaseRuntimeContext(ctx)
return copied, nil
}
func marshalIndent(v interface{}, prefix, indent string, optFuncs ...EncodeOptionFunc) ([]byte, error) {
ctx := encoder.TakeRuntimeContext()
ctx.Option.Flag = 0
ctx.Option.Flag |= (encoder.HTMLEscapeOption | encoder.IndentOption)
for _, optFunc := range optFuncs {
optFunc(ctx.Option)
}
buf, err := encodeIndent(ctx, v, prefix, indent)
if err != nil {
encoder.ReleaseRuntimeContext(ctx)
return nil, err
}
buf = buf[:len(buf)-2]
copied := make([]byte, len(buf))
copy(copied, buf)
encoder.ReleaseRuntimeContext(ctx)
return copied, nil
}
func encode(ctx *encoder.RuntimeContext, v interface{}) ([]byte, error) {
b := ctx.Buf[:0]
if v == nil {
b = encoder.AppendNull(ctx, b)
b = encoder.AppendComma(ctx, b)
return b, nil
}
header := (*emptyInterface)(unsafe.Pointer(&v))
typ := header.typ
typeptr := uintptr(unsafe.Pointer(typ))
codeSet, err := encoder.CompileToGetCodeSet(typeptr)
if err != nil {
return nil, err
}
p := uintptr(header.ptr)
ctx.Init(p, codeSet.CodeLength)
ctx.KeepRefs = append(ctx.KeepRefs, header.ptr)
buf, err := encodeRunCode(ctx, b, codeSet)
if err != nil {
return nil, err
}
ctx.Buf = buf
return buf, nil
}
func encodeNoEscape(ctx *encoder.RuntimeContext, v interface{}) ([]byte, error) {
b := ctx.Buf[:0]
if v == nil {
b = encoder.AppendNull(ctx, b)
b = encoder.AppendComma(ctx, b)
return b, nil
}
header := (*emptyInterface)(unsafe.Pointer(&v))
typ := header.typ
typeptr := uintptr(unsafe.Pointer(typ))
codeSet, err := encoder.CompileToGetCodeSet(typeptr)
if err != nil {
return nil, err
}
p := uintptr(header.ptr)
ctx.Init(p, codeSet.CodeLength)
buf, err := encodeRunCode(ctx, b, codeSet)
if err != nil {
return nil, err
}
ctx.Buf = buf
return buf, nil
}
func encodeIndent(ctx *encoder.RuntimeContext, v interface{}, prefix, indent string) ([]byte, error) {
b := ctx.Buf[:0]
if v == nil {
b = encoder.AppendNull(ctx, b)
b = encoder.AppendCommaIndent(ctx, b)
return b, nil
}
header := (*emptyInterface)(unsafe.Pointer(&v))
typ := header.typ
typeptr := uintptr(unsafe.Pointer(typ))
codeSet, err := encoder.CompileToGetCodeSet(typeptr)
if err != nil {
return nil, err
}
p := uintptr(header.ptr)
ctx.Init(p, codeSet.CodeLength)
buf, err := encodeRunIndentCode(ctx, b, codeSet, prefix, indent)
ctx.KeepRefs = append(ctx.KeepRefs, header.ptr)
if err != nil {
return nil, err
}
ctx.Buf = buf
return buf, nil
}
func encodeRunCode(ctx *encoder.RuntimeContext, b []byte, codeSet *encoder.OpcodeSet) ([]byte, error) {
if (ctx.Option.Flag & encoder.DebugOption) != 0 {
if (ctx.Option.Flag & encoder.ColorizeOption) != 0 {
return vm_color.DebugRun(ctx, b, codeSet)
}
return vm.DebugRun(ctx, b, codeSet)
}
if (ctx.Option.Flag & encoder.ColorizeOption) != 0 {
return vm_color.Run(ctx, b, codeSet)
}
return vm.Run(ctx, b, codeSet)
}
func encodeRunIndentCode(ctx *encoder.RuntimeContext, b []byte, codeSet *encoder.OpcodeSet, prefix, indent string) ([]byte, error) {
ctx.Prefix = []byte(prefix)
ctx.IndentStr = []byte(indent)
if (ctx.Option.Flag & encoder.DebugOption) != 0 {
if (ctx.Option.Flag & encoder.ColorizeOption) != 0 {
return vm_color_indent.DebugRun(ctx, b, codeSet)
}
return vm_indent.DebugRun(ctx, b, codeSet)
}
if (ctx.Option.Flag & encoder.ColorizeOption) != 0 {
return vm_color_indent.Run(ctx, b, codeSet)
}
return vm_indent.Run(ctx, b, codeSet)
}

39
vendor/github.com/goccy/go-json/error.go generated vendored Normal file
View File

@@ -0,0 +1,39 @@
package json
import (
"github.com/goccy/go-json/internal/errors"
)
// Before Go 1.2, an InvalidUTF8Error was returned by Marshal when
// attempting to encode a string value with invalid UTF-8 sequences.
// As of Go 1.2, Marshal instead coerces the string to valid UTF-8 by
// replacing invalid bytes with the Unicode replacement rune U+FFFD.
//
// Deprecated: No longer used; kept for compatibility.
type InvalidUTF8Error = errors.InvalidUTF8Error
// An InvalidUnmarshalError describes an invalid argument passed to Unmarshal.
// (The argument to Unmarshal must be a non-nil pointer.)
type InvalidUnmarshalError = errors.InvalidUnmarshalError
// A MarshalerError represents an error from calling a MarshalJSON or MarshalText method.
type MarshalerError = errors.MarshalerError
// A SyntaxError is a description of a JSON syntax error.
type SyntaxError = errors.SyntaxError
// An UnmarshalFieldError describes a JSON object key that
// led to an unexported (and therefore unwritable) struct field.
//
// Deprecated: No longer used; kept for compatibility.
type UnmarshalFieldError = errors.UnmarshalFieldError
// An UnmarshalTypeError describes a JSON value that was
// not appropriate for a value of a specific Go type.
type UnmarshalTypeError = errors.UnmarshalTypeError
// An UnsupportedTypeError is returned by Marshal when attempting
// to encode an unsupported value type.
type UnsupportedTypeError = errors.UnsupportedTypeError
type UnsupportedValueError = errors.UnsupportedValueError

3
vendor/github.com/goccy/go-json/go.mod generated vendored Normal file
View File

@@ -0,0 +1,3 @@
module github.com/goccy/go-json
go 1.12

0
vendor/github.com/goccy/go-json/go.sum generated vendored Normal file
View File

View File

@@ -0,0 +1,37 @@
package decoder
import (
"unsafe"
"github.com/goccy/go-json/internal/runtime"
)
type anonymousFieldDecoder struct {
structType *runtime.Type
offset uintptr
dec Decoder
}
func newAnonymousFieldDecoder(structType *runtime.Type, offset uintptr, dec Decoder) *anonymousFieldDecoder {
return &anonymousFieldDecoder{
structType: structType,
offset: offset,
dec: dec,
}
}
func (d *anonymousFieldDecoder) DecodeStream(s *Stream, depth int64, p unsafe.Pointer) error {
if *(*unsafe.Pointer)(p) == nil {
*(*unsafe.Pointer)(p) = unsafe_New(d.structType)
}
p = *(*unsafe.Pointer)(p)
return d.dec.DecodeStream(s, depth, unsafe.Pointer(uintptr(p)+d.offset))
}
func (d *anonymousFieldDecoder) Decode(ctx *RuntimeContext, cursor, depth int64, p unsafe.Pointer) (int64, error) {
if *(*unsafe.Pointer)(p) == nil {
*(*unsafe.Pointer)(p) = unsafe_New(d.structType)
}
p = *(*unsafe.Pointer)(p)
return d.dec.Decode(ctx, cursor, depth, unsafe.Pointer(uintptr(p)+d.offset))
}

View File

@@ -0,0 +1,169 @@
package decoder
import (
"unsafe"
"github.com/goccy/go-json/internal/errors"
"github.com/goccy/go-json/internal/runtime"
)
type arrayDecoder struct {
elemType *runtime.Type
size uintptr
valueDecoder Decoder
alen int
structName string
fieldName string
zeroValue unsafe.Pointer
}
func newArrayDecoder(dec Decoder, elemType *runtime.Type, alen int, structName, fieldName string) *arrayDecoder {
zeroValue := *(*unsafe.Pointer)(unsafe_New(elemType))
return &arrayDecoder{
valueDecoder: dec,
elemType: elemType,
size: elemType.Size(),
alen: alen,
structName: structName,
fieldName: fieldName,
zeroValue: zeroValue,
}
}
func (d *arrayDecoder) DecodeStream(s *Stream, depth int64, p unsafe.Pointer) error {
depth++
if depth > maxDecodeNestingDepth {
return errors.ErrExceededMaxDepth(s.char(), s.cursor)
}
for {
switch s.char() {
case ' ', '\n', '\t', '\r':
case 'n':
if err := nullBytes(s); err != nil {
return err
}
return nil
case '[':
idx := 0
s.cursor++
if s.skipWhiteSpace() == ']' {
for idx < d.alen {
*(*unsafe.Pointer)(unsafe.Pointer(uintptr(p) + uintptr(idx)*d.size)) = d.zeroValue
idx++
}
s.cursor++
return nil
}
for {
if idx < d.alen {
if err := d.valueDecoder.DecodeStream(s, depth, unsafe.Pointer(uintptr(p)+uintptr(idx)*d.size)); err != nil {
return err
}
} else {
if err := s.skipValue(depth); err != nil {
return err
}
}
idx++
switch s.skipWhiteSpace() {
case ']':
for idx < d.alen {
*(*unsafe.Pointer)(unsafe.Pointer(uintptr(p) + uintptr(idx)*d.size)) = d.zeroValue
idx++
}
s.cursor++
return nil
case ',':
s.cursor++
continue
case nul:
if s.read() {
s.cursor++
continue
}
goto ERROR
default:
goto ERROR
}
}
case nul:
if s.read() {
continue
}
goto ERROR
default:
goto ERROR
}
s.cursor++
}
ERROR:
return errors.ErrUnexpectedEndOfJSON("array", s.totalOffset())
}
func (d *arrayDecoder) Decode(ctx *RuntimeContext, cursor, depth int64, p unsafe.Pointer) (int64, error) {
buf := ctx.Buf
depth++
if depth > maxDecodeNestingDepth {
return 0, errors.ErrExceededMaxDepth(buf[cursor], cursor)
}
for {
switch buf[cursor] {
case ' ', '\n', '\t', '\r':
cursor++
continue
case 'n':
if err := validateNull(buf, cursor); err != nil {
return 0, err
}
cursor += 4
return cursor, nil
case '[':
idx := 0
cursor++
cursor = skipWhiteSpace(buf, cursor)
if buf[cursor] == ']' {
for idx < d.alen {
*(*unsafe.Pointer)(unsafe.Pointer(uintptr(p) + uintptr(idx)*d.size)) = d.zeroValue
idx++
}
cursor++
return cursor, nil
}
for {
if idx < d.alen {
c, err := d.valueDecoder.Decode(ctx, cursor, depth, unsafe.Pointer(uintptr(p)+uintptr(idx)*d.size))
if err != nil {
return 0, err
}
cursor = c
} else {
c, err := skipValue(buf, cursor, depth)
if err != nil {
return 0, err
}
cursor = c
}
idx++
cursor = skipWhiteSpace(buf, cursor)
switch buf[cursor] {
case ']':
for idx < d.alen {
*(*unsafe.Pointer)(unsafe.Pointer(uintptr(p) + uintptr(idx)*d.size)) = d.zeroValue
idx++
}
cursor++
return cursor, nil
case ',':
cursor++
continue
default:
return 0, errors.ErrInvalidCharacter(buf[cursor], "array", cursor)
}
}
default:
return 0, errors.ErrUnexpectedEndOfJSON("array", cursor)
}
}
}

View File

@@ -0,0 +1,78 @@
package decoder
import (
"unsafe"
"github.com/goccy/go-json/internal/errors"
)
type boolDecoder struct {
structName string
fieldName string
}
func newBoolDecoder(structName, fieldName string) *boolDecoder {
return &boolDecoder{structName: structName, fieldName: fieldName}
}
func (d *boolDecoder) DecodeStream(s *Stream, depth int64, p unsafe.Pointer) error {
c := s.skipWhiteSpace()
for {
switch c {
case 't':
if err := trueBytes(s); err != nil {
return err
}
**(**bool)(unsafe.Pointer(&p)) = true
return nil
case 'f':
if err := falseBytes(s); err != nil {
return err
}
**(**bool)(unsafe.Pointer(&p)) = false
return nil
case 'n':
if err := nullBytes(s); err != nil {
return err
}
return nil
case nul:
if s.read() {
c = s.char()
continue
}
goto ERROR
}
break
}
ERROR:
return errors.ErrUnexpectedEndOfJSON("bool", s.totalOffset())
}
func (d *boolDecoder) Decode(ctx *RuntimeContext, cursor, depth int64, p unsafe.Pointer) (int64, error) {
buf := ctx.Buf
cursor = skipWhiteSpace(buf, cursor)
switch buf[cursor] {
case 't':
if err := validateTrue(buf, cursor); err != nil {
return 0, err
}
cursor += 4
**(**bool)(unsafe.Pointer(&p)) = true
return cursor, nil
case 'f':
if err := validateFalse(buf, cursor); err != nil {
return 0, err
}
cursor += 5
**(**bool)(unsafe.Pointer(&p)) = false
return cursor, nil
case 'n':
if err := validateNull(buf, cursor); err != nil {
return 0, err
}
cursor += 4
return cursor, nil
}
return 0, errors.ErrUnexpectedEndOfJSON("bool", cursor)
}

View File

@@ -0,0 +1,177 @@
package decoder
import (
"encoding/base64"
"unsafe"
"github.com/goccy/go-json/internal/errors"
"github.com/goccy/go-json/internal/runtime"
)
type bytesDecoder struct {
typ *runtime.Type
sliceDecoder Decoder
structName string
fieldName string
}
func byteUnmarshalerSliceDecoder(typ *runtime.Type, structName string, fieldName string) Decoder {
var unmarshalDecoder Decoder
switch {
case runtime.PtrTo(typ).Implements(unmarshalJSONType):
unmarshalDecoder = newUnmarshalJSONDecoder(runtime.PtrTo(typ), structName, fieldName)
case runtime.PtrTo(typ).Implements(unmarshalTextType):
unmarshalDecoder = newUnmarshalTextDecoder(runtime.PtrTo(typ), structName, fieldName)
}
if unmarshalDecoder == nil {
return nil
}
return newSliceDecoder(unmarshalDecoder, typ, 1, structName, fieldName)
}
func newBytesDecoder(typ *runtime.Type, structName string, fieldName string) *bytesDecoder {
return &bytesDecoder{
typ: typ,
sliceDecoder: byteUnmarshalerSliceDecoder(typ, structName, fieldName),
structName: structName,
fieldName: fieldName,
}
}
func (d *bytesDecoder) DecodeStream(s *Stream, depth int64, p unsafe.Pointer) error {
bytes, err := d.decodeStreamBinary(s, depth, p)
if err != nil {
return err
}
if bytes == nil {
s.reset()
return nil
}
decodedLen := base64.StdEncoding.DecodedLen(len(bytes))
buf := make([]byte, decodedLen)
n, err := base64.StdEncoding.Decode(buf, bytes)
if err != nil {
return err
}
*(*[]byte)(p) = buf[:n]
s.reset()
return nil
}
func (d *bytesDecoder) Decode(ctx *RuntimeContext, cursor, depth int64, p unsafe.Pointer) (int64, error) {
bytes, c, err := d.decodeBinary(ctx, cursor, depth, p)
if err != nil {
return 0, err
}
if bytes == nil {
return c, nil
}
cursor = c
decodedLen := base64.StdEncoding.DecodedLen(len(bytes))
b := make([]byte, decodedLen)
n, err := base64.StdEncoding.Decode(b, bytes)
if err != nil {
return 0, err
}
*(*[]byte)(p) = b[:n]
return cursor, nil
}
func binaryBytes(s *Stream) ([]byte, error) {
s.cursor++
start := s.cursor
for {
switch s.char() {
case '"':
literal := s.buf[start:s.cursor]
s.cursor++
return literal, nil
case nul:
if s.read() {
continue
}
goto ERROR
}
s.cursor++
}
ERROR:
return nil, errors.ErrUnexpectedEndOfJSON("[]byte", s.totalOffset())
}
func (d *bytesDecoder) decodeStreamBinary(s *Stream, depth int64, p unsafe.Pointer) ([]byte, error) {
for {
switch s.char() {
case ' ', '\n', '\t', '\r':
s.cursor++
continue
case '"':
return binaryBytes(s)
case 'n':
if err := nullBytes(s); err != nil {
return nil, err
}
return nil, nil
case '[':
if d.sliceDecoder == nil {
return nil, &errors.UnmarshalTypeError{
Type: runtime.RType2Type(d.typ),
Offset: s.totalOffset(),
}
}
if err := d.sliceDecoder.DecodeStream(s, depth, p); err != nil {
return nil, err
}
return nil, nil
case nul:
if s.read() {
continue
}
}
break
}
return nil, errors.ErrNotAtBeginningOfValue(s.totalOffset())
}
func (d *bytesDecoder) decodeBinary(ctx *RuntimeContext, cursor, depth int64, p unsafe.Pointer) ([]byte, int64, error) {
buf := ctx.Buf
for {
switch buf[cursor] {
case ' ', '\n', '\t', '\r':
cursor++
case '"':
cursor++
start := cursor
for {
switch buf[cursor] {
case '"':
literal := buf[start:cursor]
cursor++
return literal, cursor, nil
case nul:
return nil, 0, errors.ErrUnexpectedEndOfJSON("[]byte", cursor)
}
cursor++
}
case '[':
if d.sliceDecoder == nil {
return nil, 0, &errors.UnmarshalTypeError{
Type: runtime.RType2Type(d.typ),
Offset: cursor,
}
}
c, err := d.sliceDecoder.Decode(ctx, cursor, depth, p)
if err != nil {
return nil, 0, err
}
return nil, c, nil
case 'n':
if err := validateNull(buf, cursor); err != nil {
return nil, 0, err
}
cursor += 4
return nil, cursor, nil
default:
return nil, 0, errors.ErrNotAtBeginningOfValue(cursor)
}
}
}

View File

@@ -0,0 +1,510 @@
package decoder
import (
"encoding/json"
"fmt"
"reflect"
"strings"
"sync/atomic"
"unicode"
"unsafe"
"github.com/goccy/go-json/internal/errors"
"github.com/goccy/go-json/internal/runtime"
)
var (
jsonNumberType = reflect.TypeOf(json.Number(""))
typeAddr *runtime.TypeAddr
cachedDecoderMap unsafe.Pointer // map[uintptr]decoder
cachedDecoder []Decoder
)
func init() {
typeAddr = runtime.AnalyzeTypeAddr()
if typeAddr == nil {
typeAddr = &runtime.TypeAddr{}
}
cachedDecoder = make([]Decoder, typeAddr.AddrRange>>typeAddr.AddrShift)
}
func loadDecoderMap() map[uintptr]Decoder {
p := atomic.LoadPointer(&cachedDecoderMap)
return *(*map[uintptr]Decoder)(unsafe.Pointer(&p))
}
func storeDecoder(typ uintptr, dec Decoder, m map[uintptr]Decoder) {
newDecoderMap := make(map[uintptr]Decoder, len(m)+1)
newDecoderMap[typ] = dec
for k, v := range m {
newDecoderMap[k] = v
}
atomic.StorePointer(&cachedDecoderMap, *(*unsafe.Pointer)(unsafe.Pointer(&newDecoderMap)))
}
func compileToGetDecoderSlowPath(typeptr uintptr, typ *runtime.Type) (Decoder, error) {
decoderMap := loadDecoderMap()
if dec, exists := decoderMap[typeptr]; exists {
return dec, nil
}
dec, err := compileHead(typ, map[uintptr]Decoder{})
if err != nil {
return nil, err
}
storeDecoder(typeptr, dec, decoderMap)
return dec, nil
}
func compileHead(typ *runtime.Type, structTypeToDecoder map[uintptr]Decoder) (Decoder, error) {
switch {
case implementsUnmarshalJSONType(runtime.PtrTo(typ)):
return newUnmarshalJSONDecoder(runtime.PtrTo(typ), "", ""), nil
case runtime.PtrTo(typ).Implements(unmarshalTextType):
return newUnmarshalTextDecoder(runtime.PtrTo(typ), "", ""), nil
}
return compile(typ.Elem(), "", "", structTypeToDecoder)
}
func compile(typ *runtime.Type, structName, fieldName string, structTypeToDecoder map[uintptr]Decoder) (Decoder, error) {
switch {
case implementsUnmarshalJSONType(runtime.PtrTo(typ)):
return newUnmarshalJSONDecoder(runtime.PtrTo(typ), structName, fieldName), nil
case runtime.PtrTo(typ).Implements(unmarshalTextType):
return newUnmarshalTextDecoder(runtime.PtrTo(typ), structName, fieldName), nil
}
switch typ.Kind() {
case reflect.Ptr:
return compilePtr(typ, structName, fieldName, structTypeToDecoder)
case reflect.Struct:
return compileStruct(typ, structName, fieldName, structTypeToDecoder)
case reflect.Slice:
elem := typ.Elem()
if elem.Kind() == reflect.Uint8 {
return compileBytes(elem, structName, fieldName)
}
return compileSlice(typ, structName, fieldName, structTypeToDecoder)
case reflect.Array:
return compileArray(typ, structName, fieldName, structTypeToDecoder)
case reflect.Map:
return compileMap(typ, structName, fieldName, structTypeToDecoder)
case reflect.Interface:
return compileInterface(typ, structName, fieldName)
case reflect.Uintptr:
return compileUint(typ, structName, fieldName)
case reflect.Int:
return compileInt(typ, structName, fieldName)
case reflect.Int8:
return compileInt8(typ, structName, fieldName)
case reflect.Int16:
return compileInt16(typ, structName, fieldName)
case reflect.Int32:
return compileInt32(typ, structName, fieldName)
case reflect.Int64:
return compileInt64(typ, structName, fieldName)
case reflect.Uint:
return compileUint(typ, structName, fieldName)
case reflect.Uint8:
return compileUint8(typ, structName, fieldName)
case reflect.Uint16:
return compileUint16(typ, structName, fieldName)
case reflect.Uint32:
return compileUint32(typ, structName, fieldName)
case reflect.Uint64:
return compileUint64(typ, structName, fieldName)
case reflect.String:
return compileString(typ, structName, fieldName)
case reflect.Bool:
return compileBool(structName, fieldName)
case reflect.Float32:
return compileFloat32(structName, fieldName)
case reflect.Float64:
return compileFloat64(structName, fieldName)
case reflect.Func:
return compileFunc(typ, structName, fieldName)
}
return nil, &errors.UnmarshalTypeError{
Value: "object",
Type: runtime.RType2Type(typ),
Offset: 0,
Struct: structName,
Field: fieldName,
}
}
func isStringTagSupportedType(typ *runtime.Type) bool {
switch {
case implementsUnmarshalJSONType(runtime.PtrTo(typ)):
return false
case runtime.PtrTo(typ).Implements(unmarshalTextType):
return false
}
switch typ.Kind() {
case reflect.Map:
return false
case reflect.Slice:
return false
case reflect.Array:
return false
case reflect.Struct:
return false
case reflect.Interface:
return false
}
return true
}
func compileMapKey(typ *runtime.Type, structName, fieldName string, structTypeToDecoder map[uintptr]Decoder) (Decoder, error) {
if runtime.PtrTo(typ).Implements(unmarshalTextType) {
return newUnmarshalTextDecoder(runtime.PtrTo(typ), structName, fieldName), nil
}
dec, err := compile(typ, structName, fieldName, structTypeToDecoder)
if err != nil {
return nil, err
}
for {
switch t := dec.(type) {
case *stringDecoder, *interfaceDecoder:
return dec, nil
case *boolDecoder, *intDecoder, *uintDecoder, *numberDecoder:
return newWrappedStringDecoder(typ, dec, structName, fieldName), nil
case *ptrDecoder:
dec = t.dec
default:
goto ERROR
}
}
ERROR:
return nil, &errors.UnmarshalTypeError{
Value: "object",
Type: runtime.RType2Type(typ),
Offset: 0,
Struct: structName,
Field: fieldName,
}
}
func compilePtr(typ *runtime.Type, structName, fieldName string, structTypeToDecoder map[uintptr]Decoder) (Decoder, error) {
dec, err := compile(typ.Elem(), structName, fieldName, structTypeToDecoder)
if err != nil {
return nil, err
}
return newPtrDecoder(dec, typ.Elem(), structName, fieldName), nil
}
func compileInt(typ *runtime.Type, structName, fieldName string) (Decoder, error) {
return newIntDecoder(typ, structName, fieldName, func(p unsafe.Pointer, v int64) {
*(*int)(p) = int(v)
}), nil
}
func compileInt8(typ *runtime.Type, structName, fieldName string) (Decoder, error) {
return newIntDecoder(typ, structName, fieldName, func(p unsafe.Pointer, v int64) {
*(*int8)(p) = int8(v)
}), nil
}
func compileInt16(typ *runtime.Type, structName, fieldName string) (Decoder, error) {
return newIntDecoder(typ, structName, fieldName, func(p unsafe.Pointer, v int64) {
*(*int16)(p) = int16(v)
}), nil
}
func compileInt32(typ *runtime.Type, structName, fieldName string) (Decoder, error) {
return newIntDecoder(typ, structName, fieldName, func(p unsafe.Pointer, v int64) {
*(*int32)(p) = int32(v)
}), nil
}
func compileInt64(typ *runtime.Type, structName, fieldName string) (Decoder, error) {
return newIntDecoder(typ, structName, fieldName, func(p unsafe.Pointer, v int64) {
*(*int64)(p) = v
}), nil
}
func compileUint(typ *runtime.Type, structName, fieldName string) (Decoder, error) {
return newUintDecoder(typ, structName, fieldName, func(p unsafe.Pointer, v uint64) {
*(*uint)(p) = uint(v)
}), nil
}
func compileUint8(typ *runtime.Type, structName, fieldName string) (Decoder, error) {
return newUintDecoder(typ, structName, fieldName, func(p unsafe.Pointer, v uint64) {
*(*uint8)(p) = uint8(v)
}), nil
}
func compileUint16(typ *runtime.Type, structName, fieldName string) (Decoder, error) {
return newUintDecoder(typ, structName, fieldName, func(p unsafe.Pointer, v uint64) {
*(*uint16)(p) = uint16(v)
}), nil
}
func compileUint32(typ *runtime.Type, structName, fieldName string) (Decoder, error) {
return newUintDecoder(typ, structName, fieldName, func(p unsafe.Pointer, v uint64) {
*(*uint32)(p) = uint32(v)
}), nil
}
func compileUint64(typ *runtime.Type, structName, fieldName string) (Decoder, error) {
return newUintDecoder(typ, structName, fieldName, func(p unsafe.Pointer, v uint64) {
*(*uint64)(p) = v
}), nil
}
func compileFloat32(structName, fieldName string) (Decoder, error) {
return newFloatDecoder(structName, fieldName, func(p unsafe.Pointer, v float64) {
*(*float32)(p) = float32(v)
}), nil
}
func compileFloat64(structName, fieldName string) (Decoder, error) {
return newFloatDecoder(structName, fieldName, func(p unsafe.Pointer, v float64) {
*(*float64)(p) = v
}), nil
}
func compileString(typ *runtime.Type, structName, fieldName string) (Decoder, error) {
if typ == runtime.Type2RType(jsonNumberType) {
return newNumberDecoder(structName, fieldName, func(p unsafe.Pointer, v json.Number) {
*(*json.Number)(p) = v
}), nil
}
return newStringDecoder(structName, fieldName), nil
}
func compileBool(structName, fieldName string) (Decoder, error) {
return newBoolDecoder(structName, fieldName), nil
}
func compileBytes(typ *runtime.Type, structName, fieldName string) (Decoder, error) {
return newBytesDecoder(typ, structName, fieldName), nil
}
func compileSlice(typ *runtime.Type, structName, fieldName string, structTypeToDecoder map[uintptr]Decoder) (Decoder, error) {
elem := typ.Elem()
decoder, err := compile(elem, structName, fieldName, structTypeToDecoder)
if err != nil {
return nil, err
}
return newSliceDecoder(decoder, elem, elem.Size(), structName, fieldName), nil
}
func compileArray(typ *runtime.Type, structName, fieldName string, structTypeToDecoder map[uintptr]Decoder) (Decoder, error) {
elem := typ.Elem()
decoder, err := compile(elem, structName, fieldName, structTypeToDecoder)
if err != nil {
return nil, err
}
return newArrayDecoder(decoder, elem, typ.Len(), structName, fieldName), nil
}
func compileMap(typ *runtime.Type, structName, fieldName string, structTypeToDecoder map[uintptr]Decoder) (Decoder, error) {
keyDec, err := compileMapKey(typ.Key(), structName, fieldName, structTypeToDecoder)
if err != nil {
return nil, err
}
valueDec, err := compile(typ.Elem(), structName, fieldName, structTypeToDecoder)
if err != nil {
return nil, err
}
return newMapDecoder(typ, typ.Key(), keyDec, typ.Elem(), valueDec, structName, fieldName), nil
}
func compileInterface(typ *runtime.Type, structName, fieldName string) (Decoder, error) {
return newInterfaceDecoder(typ, structName, fieldName), nil
}
func compileFunc(typ *runtime.Type, strutName, fieldName string) (Decoder, error) {
return newFuncDecoder(typ, strutName, fieldName), nil
}
func removeConflictFields(fieldMap map[string]*structFieldSet, conflictedMap map[string]struct{}, dec *structDecoder, field reflect.StructField) {
for k, v := range dec.fieldMap {
if _, exists := conflictedMap[k]; exists {
// already conflicted key
continue
}
set, exists := fieldMap[k]
if !exists {
fieldSet := &structFieldSet{
dec: v.dec,
offset: field.Offset + v.offset,
isTaggedKey: v.isTaggedKey,
key: k,
keyLen: int64(len(k)),
}
fieldMap[k] = fieldSet
lower := strings.ToLower(k)
if _, exists := fieldMap[lower]; !exists {
fieldMap[lower] = fieldSet
}
continue
}
if set.isTaggedKey {
if v.isTaggedKey {
// conflict tag key
delete(fieldMap, k)
delete(fieldMap, strings.ToLower(k))
conflictedMap[k] = struct{}{}
conflictedMap[strings.ToLower(k)] = struct{}{}
}
} else {
if v.isTaggedKey {
fieldSet := &structFieldSet{
dec: v.dec,
offset: field.Offset + v.offset,
isTaggedKey: v.isTaggedKey,
key: k,
keyLen: int64(len(k)),
}
fieldMap[k] = fieldSet
lower := strings.ToLower(k)
if _, exists := fieldMap[lower]; !exists {
fieldMap[lower] = fieldSet
}
} else {
// conflict tag key
delete(fieldMap, k)
delete(fieldMap, strings.ToLower(k))
conflictedMap[k] = struct{}{}
conflictedMap[strings.ToLower(k)] = struct{}{}
}
}
}
}
func compileStruct(typ *runtime.Type, structName, fieldName string, structTypeToDecoder map[uintptr]Decoder) (Decoder, error) {
fieldNum := typ.NumField()
conflictedMap := map[string]struct{}{}
fieldMap := map[string]*structFieldSet{}
typeptr := uintptr(unsafe.Pointer(typ))
if dec, exists := structTypeToDecoder[typeptr]; exists {
return dec, nil
}
structDec := newStructDecoder(structName, fieldName, fieldMap)
structTypeToDecoder[typeptr] = structDec
structName = typ.Name()
for i := 0; i < fieldNum; i++ {
field := typ.Field(i)
if runtime.IsIgnoredStructField(field) {
continue
}
isUnexportedField := unicode.IsLower([]rune(field.Name)[0])
tag := runtime.StructTagFromField(field)
dec, err := compile(runtime.Type2RType(field.Type), structName, field.Name, structTypeToDecoder)
if err != nil {
return nil, err
}
if field.Anonymous && !tag.IsTaggedKey {
if stDec, ok := dec.(*structDecoder); ok {
if runtime.Type2RType(field.Type) == typ {
// recursive definition
continue
}
removeConflictFields(fieldMap, conflictedMap, stDec, field)
} else if pdec, ok := dec.(*ptrDecoder); ok {
contentDec := pdec.contentDecoder()
if pdec.typ == typ {
// recursive definition
continue
}
var fieldSetErr error
if isUnexportedField {
fieldSetErr = fmt.Errorf(
"json: cannot set embedded pointer to unexported struct: %v",
field.Type.Elem(),
)
}
if dec, ok := contentDec.(*structDecoder); ok {
for k, v := range dec.fieldMap {
if _, exists := conflictedMap[k]; exists {
// already conflicted key
continue
}
set, exists := fieldMap[k]
if !exists {
fieldSet := &structFieldSet{
dec: newAnonymousFieldDecoder(pdec.typ, v.offset, v.dec),
offset: field.Offset,
isTaggedKey: v.isTaggedKey,
key: k,
keyLen: int64(len(k)),
err: fieldSetErr,
}
fieldMap[k] = fieldSet
lower := strings.ToLower(k)
if _, exists := fieldMap[lower]; !exists {
fieldMap[lower] = fieldSet
}
continue
}
if set.isTaggedKey {
if v.isTaggedKey {
// conflict tag key
delete(fieldMap, k)
delete(fieldMap, strings.ToLower(k))
conflictedMap[k] = struct{}{}
conflictedMap[strings.ToLower(k)] = struct{}{}
}
} else {
if v.isTaggedKey {
fieldSet := &structFieldSet{
dec: newAnonymousFieldDecoder(pdec.typ, v.offset, v.dec),
offset: field.Offset,
isTaggedKey: v.isTaggedKey,
key: k,
keyLen: int64(len(k)),
err: fieldSetErr,
}
fieldMap[k] = fieldSet
lower := strings.ToLower(k)
if _, exists := fieldMap[lower]; !exists {
fieldMap[lower] = fieldSet
}
} else {
// conflict tag key
delete(fieldMap, k)
delete(fieldMap, strings.ToLower(k))
conflictedMap[k] = struct{}{}
conflictedMap[strings.ToLower(k)] = struct{}{}
}
}
}
}
}
} else {
if tag.IsString && isStringTagSupportedType(runtime.Type2RType(field.Type)) {
dec = newWrappedStringDecoder(runtime.Type2RType(field.Type), dec, structName, field.Name)
}
var key string
if tag.Key != "" {
key = tag.Key
} else {
key = field.Name
}
fieldSet := &structFieldSet{
dec: dec,
offset: field.Offset,
isTaggedKey: tag.IsTaggedKey,
key: key,
keyLen: int64(len(key)),
}
fieldMap[key] = fieldSet
lower := strings.ToLower(key)
if _, exists := fieldMap[lower]; !exists {
fieldMap[lower] = fieldSet
}
}
}
delete(structTypeToDecoder, typeptr)
structDec.tryOptimize()
return structDec, nil
}
func implementsUnmarshalJSONType(typ *runtime.Type) bool {
return typ.Implements(unmarshalJSONType) || typ.Implements(unmarshalJSONContextType)
}

View File

@@ -0,0 +1,28 @@
// +build !race
package decoder
import (
"unsafe"
"github.com/goccy/go-json/internal/runtime"
)
func CompileToGetDecoder(typ *runtime.Type) (Decoder, error) {
typeptr := uintptr(unsafe.Pointer(typ))
if typeptr > typeAddr.MaxTypeAddr {
return compileToGetDecoderSlowPath(typeptr, typ)
}
index := (typeptr - typeAddr.BaseTypeAddr) >> typeAddr.AddrShift
if dec := cachedDecoder[index]; dec != nil {
return dec, nil
}
dec, err := compileHead(typ, map[uintptr]Decoder{})
if err != nil {
return nil, err
}
cachedDecoder[index] = dec
return dec, nil
}

View File

@@ -0,0 +1,36 @@
// +build race
package decoder
import (
"sync"
"unsafe"
"github.com/goccy/go-json/internal/runtime"
)
var decMu sync.RWMutex
func CompileToGetDecoder(typ *runtime.Type) (Decoder, error) {
typeptr := uintptr(unsafe.Pointer(typ))
if typeptr > typeAddr.MaxTypeAddr {
return compileToGetDecoderSlowPath(typeptr, typ)
}
index := (typeptr - typeAddr.BaseTypeAddr) >> typeAddr.AddrShift
decMu.RLock()
if dec := cachedDecoder[index]; dec != nil {
decMu.RUnlock()
return dec, nil
}
decMu.RUnlock()
dec, err := compileHead(typ, map[uintptr]Decoder{})
if err != nil {
return nil, err
}
decMu.Lock()
cachedDecoder[index] = dec
decMu.Unlock()
return dec, nil
}

View File

@@ -0,0 +1,254 @@
package decoder
import (
"sync"
"unsafe"
"github.com/goccy/go-json/internal/errors"
)
type RuntimeContext struct {
Buf []byte
Option *Option
}
var (
runtimeContextPool = sync.Pool{
New: func() interface{} {
return &RuntimeContext{
Option: &Option{},
}
},
}
)
func TakeRuntimeContext() *RuntimeContext {
return runtimeContextPool.Get().(*RuntimeContext)
}
func ReleaseRuntimeContext(ctx *RuntimeContext) {
runtimeContextPool.Put(ctx)
}
var (
isWhiteSpace = [256]bool{}
)
func init() {
isWhiteSpace[' '] = true
isWhiteSpace['\n'] = true
isWhiteSpace['\t'] = true
isWhiteSpace['\r'] = true
}
func char(ptr unsafe.Pointer, offset int64) byte {
return *(*byte)(unsafe.Pointer(uintptr(ptr) + uintptr(offset)))
}
func skipWhiteSpace(buf []byte, cursor int64) int64 {
for isWhiteSpace[buf[cursor]] {
cursor++
}
return cursor
}
func skipObject(buf []byte, cursor, depth int64) (int64, error) {
braceCount := 1
for {
switch buf[cursor] {
case '{':
braceCount++
depth++
if depth > maxDecodeNestingDepth {
return 0, errors.ErrExceededMaxDepth(buf[cursor], cursor)
}
case '}':
depth--
braceCount--
if braceCount == 0 {
return cursor + 1, nil
}
case '[':
depth++
if depth > maxDecodeNestingDepth {
return 0, errors.ErrExceededMaxDepth(buf[cursor], cursor)
}
case ']':
depth--
case '"':
for {
cursor++
switch buf[cursor] {
case '\\':
cursor++
if buf[cursor] == nul {
return 0, errors.ErrUnexpectedEndOfJSON("string of object", cursor)
}
case '"':
goto SWITCH_OUT
case nul:
return 0, errors.ErrUnexpectedEndOfJSON("string of object", cursor)
}
}
case nul:
return 0, errors.ErrUnexpectedEndOfJSON("object of object", cursor)
}
SWITCH_OUT:
cursor++
}
}
func skipArray(buf []byte, cursor, depth int64) (int64, error) {
bracketCount := 1
for {
switch buf[cursor] {
case '[':
bracketCount++
depth++
if depth > maxDecodeNestingDepth {
return 0, errors.ErrExceededMaxDepth(buf[cursor], cursor)
}
case ']':
bracketCount--
depth--
if bracketCount == 0 {
return cursor + 1, nil
}
case '{':
depth++
if depth > maxDecodeNestingDepth {
return 0, errors.ErrExceededMaxDepth(buf[cursor], cursor)
}
case '}':
depth--
case '"':
for {
cursor++
switch buf[cursor] {
case '\\':
cursor++
if buf[cursor] == nul {
return 0, errors.ErrUnexpectedEndOfJSON("string of object", cursor)
}
case '"':
goto SWITCH_OUT
case nul:
return 0, errors.ErrUnexpectedEndOfJSON("string of object", cursor)
}
}
case nul:
return 0, errors.ErrUnexpectedEndOfJSON("array of object", cursor)
}
SWITCH_OUT:
cursor++
}
}
func skipValue(buf []byte, cursor, depth int64) (int64, error) {
for {
switch buf[cursor] {
case ' ', '\t', '\n', '\r':
cursor++
continue
case '{':
return skipObject(buf, cursor+1, depth+1)
case '[':
return skipArray(buf, cursor+1, depth+1)
case '"':
for {
cursor++
switch buf[cursor] {
case '\\':
cursor++
if buf[cursor] == nul {
return 0, errors.ErrUnexpectedEndOfJSON("string of object", cursor)
}
case '"':
return cursor + 1, nil
case nul:
return 0, errors.ErrUnexpectedEndOfJSON("string of object", cursor)
}
}
case '-', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9':
for {
cursor++
if floatTable[buf[cursor]] {
continue
}
break
}
return cursor, nil
case 't':
if err := validateTrue(buf, cursor); err != nil {
return 0, err
}
cursor += 4
return cursor, nil
case 'f':
if err := validateFalse(buf, cursor); err != nil {
return 0, err
}
cursor += 5
return cursor, nil
case 'n':
if err := validateNull(buf, cursor); err != nil {
return 0, err
}
cursor += 4
return cursor, nil
default:
return cursor, errors.ErrUnexpectedEndOfJSON("null", cursor)
}
}
}
func validateTrue(buf []byte, cursor int64) error {
if cursor+3 >= int64(len(buf)) {
return errors.ErrUnexpectedEndOfJSON("true", cursor)
}
if buf[cursor+1] != 'r' {
return errors.ErrInvalidCharacter(buf[cursor+1], "true", cursor)
}
if buf[cursor+2] != 'u' {
return errors.ErrInvalidCharacter(buf[cursor+2], "true", cursor)
}
if buf[cursor+3] != 'e' {
return errors.ErrInvalidCharacter(buf[cursor+3], "true", cursor)
}
return nil
}
func validateFalse(buf []byte, cursor int64) error {
if cursor+4 >= int64(len(buf)) {
return errors.ErrUnexpectedEndOfJSON("false", cursor)
}
if buf[cursor+1] != 'a' {
return errors.ErrInvalidCharacter(buf[cursor+1], "false", cursor)
}
if buf[cursor+2] != 'l' {
return errors.ErrInvalidCharacter(buf[cursor+2], "false", cursor)
}
if buf[cursor+3] != 's' {
return errors.ErrInvalidCharacter(buf[cursor+3], "false", cursor)
}
if buf[cursor+4] != 'e' {
return errors.ErrInvalidCharacter(buf[cursor+4], "false", cursor)
}
return nil
}
func validateNull(buf []byte, cursor int64) error {
if cursor+3 >= int64(len(buf)) {
return errors.ErrUnexpectedEndOfJSON("null", cursor)
}
if buf[cursor+1] != 'u' {
return errors.ErrInvalidCharacter(buf[cursor+1], "null", cursor)
}
if buf[cursor+2] != 'l' {
return errors.ErrInvalidCharacter(buf[cursor+2], "null", cursor)
}
if buf[cursor+3] != 'l' {
return errors.ErrInvalidCharacter(buf[cursor+3], "null", cursor)
}
return nil
}

View File

@@ -0,0 +1,158 @@
package decoder
import (
"strconv"
"unsafe"
"github.com/goccy/go-json/internal/errors"
)
type floatDecoder struct {
op func(unsafe.Pointer, float64)
structName string
fieldName string
}
func newFloatDecoder(structName, fieldName string, op func(unsafe.Pointer, float64)) *floatDecoder {
return &floatDecoder{op: op, structName: structName, fieldName: fieldName}
}
var (
floatTable = [256]bool{
'0': true,
'1': true,
'2': true,
'3': true,
'4': true,
'5': true,
'6': true,
'7': true,
'8': true,
'9': true,
'.': true,
'e': true,
'E': true,
'+': true,
'-': true,
}
validEndNumberChar = [256]bool{
nul: true,
' ': true,
'\t': true,
'\r': true,
'\n': true,
',': true,
':': true,
'}': true,
']': true,
}
)
func floatBytes(s *Stream) []byte {
start := s.cursor
for {
s.cursor++
if floatTable[s.char()] {
continue
} else if s.char() == nul {
if s.read() {
s.cursor-- // for retry current character
continue
}
}
break
}
return s.buf[start:s.cursor]
}
func (d *floatDecoder) decodeStreamByte(s *Stream) ([]byte, error) {
for {
switch s.char() {
case ' ', '\n', '\t', '\r':
s.cursor++
continue
case '-', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9':
return floatBytes(s), nil
case 'n':
if err := nullBytes(s); err != nil {
return nil, err
}
return nil, nil
case nul:
if s.read() {
continue
}
goto ERROR
default:
goto ERROR
}
}
ERROR:
return nil, errors.ErrUnexpectedEndOfJSON("float", s.totalOffset())
}
func (d *floatDecoder) decodeByte(buf []byte, cursor int64) ([]byte, int64, error) {
for {
switch buf[cursor] {
case ' ', '\n', '\t', '\r':
cursor++
continue
case '-', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9':
start := cursor
cursor++
for floatTable[buf[cursor]] {
cursor++
}
num := buf[start:cursor]
return num, cursor, nil
case 'n':
if err := validateNull(buf, cursor); err != nil {
return nil, 0, err
}
cursor += 4
return nil, cursor, nil
default:
return nil, 0, errors.ErrUnexpectedEndOfJSON("float", cursor)
}
}
}
func (d *floatDecoder) DecodeStream(s *Stream, depth int64, p unsafe.Pointer) error {
bytes, err := d.decodeStreamByte(s)
if err != nil {
return err
}
if bytes == nil {
return nil
}
str := *(*string)(unsafe.Pointer(&bytes))
f64, err := strconv.ParseFloat(str, 64)
if err != nil {
return errors.ErrSyntax(err.Error(), s.totalOffset())
}
d.op(p, f64)
return nil
}
func (d *floatDecoder) Decode(ctx *RuntimeContext, cursor, depth int64, p unsafe.Pointer) (int64, error) {
buf := ctx.Buf
bytes, c, err := d.decodeByte(buf, cursor)
if err != nil {
return 0, err
}
if bytes == nil {
return c, nil
}
cursor = c
if !validEndNumberChar[buf[cursor]] {
return 0, errors.ErrUnexpectedEndOfJSON("float", cursor)
}
s := *(*string)(unsafe.Pointer(&bytes))
f64, err := strconv.ParseFloat(s, 64)
if err != nil {
return 0, errors.ErrSyntax(err.Error(), cursor)
}
d.op(p, f64)
return cursor, nil
}

View File

@@ -0,0 +1,141 @@
package decoder
import (
"bytes"
"unsafe"
"github.com/goccy/go-json/internal/errors"
"github.com/goccy/go-json/internal/runtime"
)
type funcDecoder struct {
typ *runtime.Type
structName string
fieldName string
}
func newFuncDecoder(typ *runtime.Type, structName, fieldName string) *funcDecoder {
fnDecoder := &funcDecoder{typ, structName, fieldName}
return fnDecoder
}
func (d *funcDecoder) DecodeStream(s *Stream, depth int64, p unsafe.Pointer) error {
s.skipWhiteSpace()
start := s.cursor
if err := s.skipValue(depth); err != nil {
return err
}
src := s.buf[start:s.cursor]
if len(src) > 0 {
switch src[0] {
case '"':
return &errors.UnmarshalTypeError{
Value: "string",
Type: runtime.RType2Type(d.typ),
Offset: s.totalOffset(),
}
case '[':
return &errors.UnmarshalTypeError{
Value: "array",
Type: runtime.RType2Type(d.typ),
Offset: s.totalOffset(),
}
case '{':
return &errors.UnmarshalTypeError{
Value: "object",
Type: runtime.RType2Type(d.typ),
Offset: s.totalOffset(),
}
case '-', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9':
return &errors.UnmarshalTypeError{
Value: "number",
Type: runtime.RType2Type(d.typ),
Offset: s.totalOffset(),
}
case 'n':
if err := nullBytes(s); err != nil {
return err
}
*(*unsafe.Pointer)(p) = nil
return nil
case 't':
if err := trueBytes(s); err == nil {
return &errors.UnmarshalTypeError{
Value: "boolean",
Type: runtime.RType2Type(d.typ),
Offset: s.totalOffset(),
}
}
case 'f':
if err := falseBytes(s); err == nil {
return &errors.UnmarshalTypeError{
Value: "boolean",
Type: runtime.RType2Type(d.typ),
Offset: s.totalOffset(),
}
}
}
}
return errors.ErrNotAtBeginningOfValue(start)
}
func (d *funcDecoder) Decode(ctx *RuntimeContext, cursor, depth int64, p unsafe.Pointer) (int64, error) {
buf := ctx.Buf
cursor = skipWhiteSpace(buf, cursor)
start := cursor
end, err := skipValue(buf, cursor, depth)
if err != nil {
return 0, err
}
src := buf[start:end]
if len(src) > 0 {
switch src[0] {
case '"':
return 0, &errors.UnmarshalTypeError{
Value: "string",
Type: runtime.RType2Type(d.typ),
Offset: start,
}
case '[':
return 0, &errors.UnmarshalTypeError{
Value: "array",
Type: runtime.RType2Type(d.typ),
Offset: start,
}
case '{':
return 0, &errors.UnmarshalTypeError{
Value: "object",
Type: runtime.RType2Type(d.typ),
Offset: start,
}
case '-', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9':
return 0, &errors.UnmarshalTypeError{
Value: "number",
Type: runtime.RType2Type(d.typ),
Offset: start,
}
case 'n':
if bytes.Equal(src, nullbytes) {
*(*unsafe.Pointer)(p) = nil
return end, nil
}
case 't':
if err := validateTrue(buf, start); err == nil {
return 0, &errors.UnmarshalTypeError{
Value: "boolean",
Type: runtime.RType2Type(d.typ),
Offset: start,
}
}
case 'f':
if err := validateFalse(buf, start); err == nil {
return 0, &errors.UnmarshalTypeError{
Value: "boolean",
Type: runtime.RType2Type(d.typ),
Offset: start,
}
}
}
}
return 0, errors.ErrNotAtBeginningOfValue(start)
}

242
vendor/github.com/goccy/go-json/internal/decoder/int.go generated vendored Normal file
View File

@@ -0,0 +1,242 @@
package decoder
import (
"fmt"
"reflect"
"unsafe"
"github.com/goccy/go-json/internal/errors"
"github.com/goccy/go-json/internal/runtime"
)
type intDecoder struct {
typ *runtime.Type
kind reflect.Kind
op func(unsafe.Pointer, int64)
structName string
fieldName string
}
func newIntDecoder(typ *runtime.Type, structName, fieldName string, op func(unsafe.Pointer, int64)) *intDecoder {
return &intDecoder{
typ: typ,
kind: typ.Kind(),
op: op,
structName: structName,
fieldName: fieldName,
}
}
func (d *intDecoder) typeError(buf []byte, offset int64) *errors.UnmarshalTypeError {
return &errors.UnmarshalTypeError{
Value: fmt.Sprintf("number %s", string(buf)),
Type: runtime.RType2Type(d.typ),
Struct: d.structName,
Field: d.fieldName,
Offset: offset,
}
}
var (
pow10i64 = [...]int64{
1e00, 1e01, 1e02, 1e03, 1e04, 1e05, 1e06, 1e07, 1e08, 1e09,
1e10, 1e11, 1e12, 1e13, 1e14, 1e15, 1e16, 1e17, 1e18,
}
pow10i64Len = len(pow10i64)
)
func (d *intDecoder) parseInt(b []byte) (int64, error) {
isNegative := false
if b[0] == '-' {
b = b[1:]
isNegative = true
}
maxDigit := len(b)
if maxDigit > pow10i64Len {
return 0, fmt.Errorf("invalid length of number")
}
sum := int64(0)
for i := 0; i < maxDigit; i++ {
c := int64(b[i]) - 48
digitValue := pow10i64[maxDigit-i-1]
sum += c * digitValue
}
if isNegative {
return -1 * sum, nil
}
return sum, nil
}
var (
numTable = [256]bool{
'0': true,
'1': true,
'2': true,
'3': true,
'4': true,
'5': true,
'6': true,
'7': true,
'8': true,
'9': true,
}
)
var (
numZeroBuf = []byte{'0'}
)
func (d *intDecoder) decodeStreamByte(s *Stream) ([]byte, error) {
for {
switch s.char() {
case ' ', '\n', '\t', '\r':
s.cursor++
continue
case '-':
start := s.cursor
for {
s.cursor++
if numTable[s.char()] {
continue
} else if s.char() == nul {
if s.read() {
s.cursor-- // for retry current character
continue
}
}
break
}
num := s.buf[start:s.cursor]
if len(num) < 2 {
goto ERROR
}
return num, nil
case '0':
s.cursor++
return numZeroBuf, nil
case '1', '2', '3', '4', '5', '6', '7', '8', '9':
start := s.cursor
for {
s.cursor++
if numTable[s.char()] {
continue
} else if s.char() == nul {
if s.read() {
s.cursor-- // for retry current character
continue
}
}
break
}
num := s.buf[start:s.cursor]
return num, nil
case 'n':
if err := nullBytes(s); err != nil {
return nil, err
}
return nil, nil
case nul:
if s.read() {
continue
}
goto ERROR
default:
return nil, d.typeError([]byte{s.char()}, s.totalOffset())
}
}
ERROR:
return nil, errors.ErrUnexpectedEndOfJSON("number(integer)", s.totalOffset())
}
func (d *intDecoder) decodeByte(buf []byte, cursor int64) ([]byte, int64, error) {
b := (*sliceHeader)(unsafe.Pointer(&buf)).data
for {
switch char(b, cursor) {
case ' ', '\n', '\t', '\r':
cursor++
continue
case '0':
cursor++
return numZeroBuf, cursor, nil
case '-', '1', '2', '3', '4', '5', '6', '7', '8', '9':
start := cursor
cursor++
for numTable[char(b, cursor)] {
cursor++
}
num := buf[start:cursor]
return num, cursor, nil
case 'n':
if err := validateNull(buf, cursor); err != nil {
return nil, 0, err
}
cursor += 4
return nil, cursor, nil
default:
return nil, 0, d.typeError([]byte{char(b, cursor)}, cursor)
}
}
}
func (d *intDecoder) DecodeStream(s *Stream, depth int64, p unsafe.Pointer) error {
bytes, err := d.decodeStreamByte(s)
if err != nil {
return err
}
if bytes == nil {
return nil
}
i64, err := d.parseInt(bytes)
if err != nil {
return d.typeError(bytes, s.totalOffset())
}
switch d.kind {
case reflect.Int8:
if i64 <= -1*(1<<7) || (1<<7) <= i64 {
return d.typeError(bytes, s.totalOffset())
}
case reflect.Int16:
if i64 <= -1*(1<<15) || (1<<15) <= i64 {
return d.typeError(bytes, s.totalOffset())
}
case reflect.Int32:
if i64 <= -1*(1<<31) || (1<<31) <= i64 {
return d.typeError(bytes, s.totalOffset())
}
}
d.op(p, i64)
s.reset()
return nil
}
func (d *intDecoder) Decode(ctx *RuntimeContext, cursor, depth int64, p unsafe.Pointer) (int64, error) {
bytes, c, err := d.decodeByte(ctx.Buf, cursor)
if err != nil {
return 0, err
}
if bytes == nil {
return c, nil
}
cursor = c
i64, err := d.parseInt(bytes)
if err != nil {
return 0, d.typeError(bytes, cursor)
}
switch d.kind {
case reflect.Int8:
if i64 <= -1*(1<<7) || (1<<7) <= i64 {
return 0, d.typeError(bytes, cursor)
}
case reflect.Int16:
if i64 <= -1*(1<<15) || (1<<15) <= i64 {
return 0, d.typeError(bytes, cursor)
}
case reflect.Int32:
if i64 <= -1*(1<<31) || (1<<31) <= i64 {
return 0, d.typeError(bytes, cursor)
}
}
d.op(p, i64)
return cursor, nil
}

View File

@@ -0,0 +1,458 @@
package decoder
import (
"bytes"
"encoding"
"encoding/json"
"reflect"
"unsafe"
"github.com/goccy/go-json/internal/errors"
"github.com/goccy/go-json/internal/runtime"
)
type interfaceDecoder struct {
typ *runtime.Type
structName string
fieldName string
sliceDecoder *sliceDecoder
mapDecoder *mapDecoder
floatDecoder *floatDecoder
numberDecoder *numberDecoder
stringDecoder *stringDecoder
}
func newEmptyInterfaceDecoder(structName, fieldName string) *interfaceDecoder {
ifaceDecoder := &interfaceDecoder{
typ: emptyInterfaceType,
structName: structName,
fieldName: fieldName,
floatDecoder: newFloatDecoder(structName, fieldName, func(p unsafe.Pointer, v float64) {
*(*interface{})(p) = v
}),
numberDecoder: newNumberDecoder(structName, fieldName, func(p unsafe.Pointer, v json.Number) {
*(*interface{})(p) = v
}),
stringDecoder: newStringDecoder(structName, fieldName),
}
ifaceDecoder.sliceDecoder = newSliceDecoder(
ifaceDecoder,
emptyInterfaceType,
emptyInterfaceType.Size(),
structName, fieldName,
)
ifaceDecoder.mapDecoder = newMapDecoder(
interfaceMapType,
stringType,
ifaceDecoder.stringDecoder,
interfaceMapType.Elem(),
ifaceDecoder,
structName,
fieldName,
)
return ifaceDecoder
}
func newInterfaceDecoder(typ *runtime.Type, structName, fieldName string) *interfaceDecoder {
emptyIfaceDecoder := newEmptyInterfaceDecoder(structName, fieldName)
stringDecoder := newStringDecoder(structName, fieldName)
return &interfaceDecoder{
typ: typ,
structName: structName,
fieldName: fieldName,
sliceDecoder: newSliceDecoder(
emptyIfaceDecoder,
emptyInterfaceType,
emptyInterfaceType.Size(),
structName, fieldName,
),
mapDecoder: newMapDecoder(
interfaceMapType,
stringType,
stringDecoder,
interfaceMapType.Elem(),
emptyIfaceDecoder,
structName,
fieldName,
),
floatDecoder: newFloatDecoder(structName, fieldName, func(p unsafe.Pointer, v float64) {
*(*interface{})(p) = v
}),
numberDecoder: newNumberDecoder(structName, fieldName, func(p unsafe.Pointer, v json.Number) {
*(*interface{})(p) = v
}),
stringDecoder: stringDecoder,
}
}
func (d *interfaceDecoder) numDecoder(s *Stream) Decoder {
if s.UseNumber {
return d.numberDecoder
}
return d.floatDecoder
}
var (
emptyInterfaceType = runtime.Type2RType(reflect.TypeOf((*interface{})(nil)).Elem())
interfaceMapType = runtime.Type2RType(
reflect.TypeOf((*map[string]interface{})(nil)).Elem(),
)
stringType = runtime.Type2RType(
reflect.TypeOf(""),
)
)
func decodeStreamUnmarshaler(s *Stream, depth int64, unmarshaler json.Unmarshaler) error {
start := s.cursor
if err := s.skipValue(depth); err != nil {
return err
}
src := s.buf[start:s.cursor]
dst := make([]byte, len(src))
copy(dst, src)
if err := unmarshaler.UnmarshalJSON(dst); err != nil {
return err
}
return nil
}
func decodeStreamUnmarshalerContext(s *Stream, depth int64, unmarshaler unmarshalerContext) error {
start := s.cursor
if err := s.skipValue(depth); err != nil {
return err
}
src := s.buf[start:s.cursor]
dst := make([]byte, len(src))
copy(dst, src)
if err := unmarshaler.UnmarshalJSON(s.Option.Context, dst); err != nil {
return err
}
return nil
}
func decodeUnmarshaler(buf []byte, cursor, depth int64, unmarshaler json.Unmarshaler) (int64, error) {
cursor = skipWhiteSpace(buf, cursor)
start := cursor
end, err := skipValue(buf, cursor, depth)
if err != nil {
return 0, err
}
src := buf[start:end]
dst := make([]byte, len(src))
copy(dst, src)
if err := unmarshaler.UnmarshalJSON(dst); err != nil {
return 0, err
}
return end, nil
}
func decodeUnmarshalerContext(ctx *RuntimeContext, buf []byte, cursor, depth int64, unmarshaler unmarshalerContext) (int64, error) {
cursor = skipWhiteSpace(buf, cursor)
start := cursor
end, err := skipValue(buf, cursor, depth)
if err != nil {
return 0, err
}
src := buf[start:end]
dst := make([]byte, len(src))
copy(dst, src)
if err := unmarshaler.UnmarshalJSON(ctx.Option.Context, dst); err != nil {
return 0, err
}
return end, nil
}
func decodeStreamTextUnmarshaler(s *Stream, depth int64, unmarshaler encoding.TextUnmarshaler, p unsafe.Pointer) error {
start := s.cursor
if err := s.skipValue(depth); err != nil {
return err
}
src := s.buf[start:s.cursor]
if bytes.Equal(src, nullbytes) {
*(*unsafe.Pointer)(p) = nil
return nil
}
dst := make([]byte, len(src))
copy(dst, src)
if err := unmarshaler.UnmarshalText(dst); err != nil {
return err
}
return nil
}
func decodeTextUnmarshaler(buf []byte, cursor, depth int64, unmarshaler encoding.TextUnmarshaler, p unsafe.Pointer) (int64, error) {
cursor = skipWhiteSpace(buf, cursor)
start := cursor
end, err := skipValue(buf, cursor, depth)
if err != nil {
return 0, err
}
src := buf[start:end]
if bytes.Equal(src, nullbytes) {
*(*unsafe.Pointer)(p) = nil
return end, nil
}
if s, ok := unquoteBytes(src); ok {
src = s
}
if err := unmarshaler.UnmarshalText(src); err != nil {
return 0, err
}
return end, nil
}
func (d *interfaceDecoder) decodeStreamEmptyInterface(s *Stream, depth int64, p unsafe.Pointer) error {
c := s.skipWhiteSpace()
for {
switch c {
case '{':
var v map[string]interface{}
ptr := unsafe.Pointer(&v)
if err := d.mapDecoder.DecodeStream(s, depth, ptr); err != nil {
return err
}
*(*interface{})(p) = v
return nil
case '[':
var v []interface{}
ptr := unsafe.Pointer(&v)
if err := d.sliceDecoder.DecodeStream(s, depth, ptr); err != nil {
return err
}
*(*interface{})(p) = v
return nil
case '-', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9':
return d.numDecoder(s).DecodeStream(s, depth, p)
case '"':
s.cursor++
start := s.cursor
for {
switch s.char() {
case '\\':
if _, err := decodeEscapeString(s, nil); err != nil {
return err
}
case '"':
literal := s.buf[start:s.cursor]
s.cursor++
*(*interface{})(p) = string(literal)
return nil
case nul:
if s.read() {
continue
}
return errors.ErrUnexpectedEndOfJSON("string", s.totalOffset())
}
s.cursor++
}
case 't':
if err := trueBytes(s); err != nil {
return err
}
**(**interface{})(unsafe.Pointer(&p)) = true
return nil
case 'f':
if err := falseBytes(s); err != nil {
return err
}
**(**interface{})(unsafe.Pointer(&p)) = false
return nil
case 'n':
if err := nullBytes(s); err != nil {
return err
}
*(*interface{})(p) = nil
return nil
case nul:
if s.read() {
c = s.char()
continue
}
}
break
}
return errors.ErrNotAtBeginningOfValue(s.totalOffset())
}
type emptyInterface struct {
typ *runtime.Type
ptr unsafe.Pointer
}
func (d *interfaceDecoder) DecodeStream(s *Stream, depth int64, p unsafe.Pointer) error {
runtimeInterfaceValue := *(*interface{})(unsafe.Pointer(&emptyInterface{
typ: d.typ,
ptr: p,
}))
rv := reflect.ValueOf(runtimeInterfaceValue)
if rv.NumMethod() > 0 && rv.CanInterface() {
if u, ok := rv.Interface().(unmarshalerContext); ok {
return decodeStreamUnmarshalerContext(s, depth, u)
}
if u, ok := rv.Interface().(json.Unmarshaler); ok {
return decodeStreamUnmarshaler(s, depth, u)
}
if u, ok := rv.Interface().(encoding.TextUnmarshaler); ok {
return decodeStreamTextUnmarshaler(s, depth, u, p)
}
if s.skipWhiteSpace() == 'n' {
if err := nullBytes(s); err != nil {
return err
}
*(*interface{})(p) = nil
return nil
}
return d.errUnmarshalType(rv.Type(), s.totalOffset())
}
iface := rv.Interface()
ifaceHeader := (*emptyInterface)(unsafe.Pointer(&iface))
typ := ifaceHeader.typ
if ifaceHeader.ptr == nil || d.typ == typ || typ == nil {
// concrete type is empty interface
return d.decodeStreamEmptyInterface(s, depth, p)
}
if typ.Kind() == reflect.Ptr && typ.Elem() == d.typ || typ.Kind() != reflect.Ptr {
return d.decodeStreamEmptyInterface(s, depth, p)
}
if s.skipWhiteSpace() == 'n' {
if err := nullBytes(s); err != nil {
return err
}
*(*interface{})(p) = nil
return nil
}
decoder, err := CompileToGetDecoder(typ)
if err != nil {
return err
}
return decoder.DecodeStream(s, depth, ifaceHeader.ptr)
}
func (d *interfaceDecoder) errUnmarshalType(typ reflect.Type, offset int64) *errors.UnmarshalTypeError {
return &errors.UnmarshalTypeError{
Value: typ.String(),
Type: typ,
Offset: offset,
Struct: d.structName,
Field: d.fieldName,
}
}
func (d *interfaceDecoder) Decode(ctx *RuntimeContext, cursor, depth int64, p unsafe.Pointer) (int64, error) {
buf := ctx.Buf
runtimeInterfaceValue := *(*interface{})(unsafe.Pointer(&emptyInterface{
typ: d.typ,
ptr: p,
}))
rv := reflect.ValueOf(runtimeInterfaceValue)
if rv.NumMethod() > 0 && rv.CanInterface() {
if u, ok := rv.Interface().(unmarshalerContext); ok {
return decodeUnmarshalerContext(ctx, buf, cursor, depth, u)
}
if u, ok := rv.Interface().(json.Unmarshaler); ok {
return decodeUnmarshaler(buf, cursor, depth, u)
}
if u, ok := rv.Interface().(encoding.TextUnmarshaler); ok {
return decodeTextUnmarshaler(buf, cursor, depth, u, p)
}
cursor = skipWhiteSpace(buf, cursor)
if buf[cursor] == 'n' {
if err := validateNull(buf, cursor); err != nil {
return 0, err
}
cursor += 4
**(**interface{})(unsafe.Pointer(&p)) = nil
return cursor, nil
}
return 0, d.errUnmarshalType(rv.Type(), cursor)
}
iface := rv.Interface()
ifaceHeader := (*emptyInterface)(unsafe.Pointer(&iface))
typ := ifaceHeader.typ
if ifaceHeader.ptr == nil || d.typ == typ || typ == nil {
// concrete type is empty interface
return d.decodeEmptyInterface(ctx, cursor, depth, p)
}
if typ.Kind() == reflect.Ptr && typ.Elem() == d.typ || typ.Kind() != reflect.Ptr {
return d.decodeEmptyInterface(ctx, cursor, depth, p)
}
cursor = skipWhiteSpace(buf, cursor)
if buf[cursor] == 'n' {
if err := validateNull(buf, cursor); err != nil {
return 0, err
}
cursor += 4
**(**interface{})(unsafe.Pointer(&p)) = nil
return cursor, nil
}
decoder, err := CompileToGetDecoder(typ)
if err != nil {
return 0, err
}
return decoder.Decode(ctx, cursor, depth, ifaceHeader.ptr)
}
func (d *interfaceDecoder) decodeEmptyInterface(ctx *RuntimeContext, cursor, depth int64, p unsafe.Pointer) (int64, error) {
buf := ctx.Buf
cursor = skipWhiteSpace(buf, cursor)
switch buf[cursor] {
case '{':
var v map[string]interface{}
ptr := unsafe.Pointer(&v)
cursor, err := d.mapDecoder.Decode(ctx, cursor, depth, ptr)
if err != nil {
return 0, err
}
**(**interface{})(unsafe.Pointer(&p)) = v
return cursor, nil
case '[':
var v []interface{}
ptr := unsafe.Pointer(&v)
cursor, err := d.sliceDecoder.Decode(ctx, cursor, depth, ptr)
if err != nil {
return 0, err
}
**(**interface{})(unsafe.Pointer(&p)) = v
return cursor, nil
case '-', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9':
return d.floatDecoder.Decode(ctx, cursor, depth, p)
case '"':
var v string
ptr := unsafe.Pointer(&v)
cursor, err := d.stringDecoder.Decode(ctx, cursor, depth, ptr)
if err != nil {
return 0, err
}
**(**interface{})(unsafe.Pointer(&p)) = v
return cursor, nil
case 't':
if err := validateTrue(buf, cursor); err != nil {
return 0, err
}
cursor += 4
**(**interface{})(unsafe.Pointer(&p)) = true
return cursor, nil
case 'f':
if err := validateFalse(buf, cursor); err != nil {
return 0, err
}
cursor += 5
**(**interface{})(unsafe.Pointer(&p)) = false
return cursor, nil
case 'n':
if err := validateNull(buf, cursor); err != nil {
return 0, err
}
cursor += 4
**(**interface{})(unsafe.Pointer(&p)) = nil
return cursor, nil
}
return cursor, errors.ErrNotAtBeginningOfValue(cursor)
}

173
vendor/github.com/goccy/go-json/internal/decoder/map.go generated vendored Normal file
View File

@@ -0,0 +1,173 @@
package decoder
import (
"reflect"
"unsafe"
"github.com/goccy/go-json/internal/errors"
"github.com/goccy/go-json/internal/runtime"
)
type mapDecoder struct {
mapType *runtime.Type
keyType *runtime.Type
valueType *runtime.Type
stringKeyType bool
keyDecoder Decoder
valueDecoder Decoder
structName string
fieldName string
}
func newMapDecoder(mapType *runtime.Type, keyType *runtime.Type, keyDec Decoder, valueType *runtime.Type, valueDec Decoder, structName, fieldName string) *mapDecoder {
return &mapDecoder{
mapType: mapType,
keyDecoder: keyDec,
keyType: keyType,
stringKeyType: keyType.Kind() == reflect.String,
valueType: valueType,
valueDecoder: valueDec,
structName: structName,
fieldName: fieldName,
}
}
//go:linkname makemap reflect.makemap
func makemap(*runtime.Type, int) unsafe.Pointer
//nolint:golint
//go:linkname mapassign_faststr runtime.mapassign_faststr
//go:noescape
func mapassign_faststr(t *runtime.Type, m unsafe.Pointer, s string) unsafe.Pointer
//go:linkname mapassign reflect.mapassign
//go:noescape
func mapassign(t *runtime.Type, m unsafe.Pointer, k, v unsafe.Pointer)
func (d *mapDecoder) mapassign(t *runtime.Type, m, k, v unsafe.Pointer) {
if d.stringKeyType {
mapV := mapassign_faststr(d.mapType, m, *(*string)(k))
typedmemmove(d.valueType, mapV, v)
} else {
mapassign(t, m, k, v)
}
}
func (d *mapDecoder) DecodeStream(s *Stream, depth int64, p unsafe.Pointer) error {
depth++
if depth > maxDecodeNestingDepth {
return errors.ErrExceededMaxDepth(s.char(), s.cursor)
}
switch s.skipWhiteSpace() {
case 'n':
if err := nullBytes(s); err != nil {
return err
}
**(**unsafe.Pointer)(unsafe.Pointer(&p)) = nil
return nil
case '{':
default:
return errors.ErrExpected("{ character for map value", s.totalOffset())
}
mapValue := *(*unsafe.Pointer)(p)
if mapValue == nil {
mapValue = makemap(d.mapType, 0)
}
if s.buf[s.cursor+1] == '}' {
*(*unsafe.Pointer)(p) = mapValue
s.cursor += 2
return nil
}
for {
s.cursor++
k := unsafe_New(d.keyType)
if err := d.keyDecoder.DecodeStream(s, depth, k); err != nil {
return err
}
s.skipWhiteSpace()
if !s.equalChar(':') {
return errors.ErrExpected("colon after object key", s.totalOffset())
}
s.cursor++
v := unsafe_New(d.valueType)
if err := d.valueDecoder.DecodeStream(s, depth, v); err != nil {
return err
}
d.mapassign(d.mapType, mapValue, k, v)
s.skipWhiteSpace()
if s.equalChar('}') {
**(**unsafe.Pointer)(unsafe.Pointer(&p)) = mapValue
s.cursor++
return nil
}
if !s.equalChar(',') {
return errors.ErrExpected("comma after object value", s.totalOffset())
}
}
}
func (d *mapDecoder) Decode(ctx *RuntimeContext, cursor, depth int64, p unsafe.Pointer) (int64, error) {
buf := ctx.Buf
depth++
if depth > maxDecodeNestingDepth {
return 0, errors.ErrExceededMaxDepth(buf[cursor], cursor)
}
cursor = skipWhiteSpace(buf, cursor)
buflen := int64(len(buf))
if buflen < 2 {
return 0, errors.ErrExpected("{} for map", cursor)
}
switch buf[cursor] {
case 'n':
if err := validateNull(buf, cursor); err != nil {
return 0, err
}
cursor += 4
**(**unsafe.Pointer)(unsafe.Pointer(&p)) = nil
return cursor, nil
case '{':
default:
return 0, errors.ErrExpected("{ character for map value", cursor)
}
cursor++
cursor = skipWhiteSpace(buf, cursor)
mapValue := *(*unsafe.Pointer)(p)
if mapValue == nil {
mapValue = makemap(d.mapType, 0)
}
if buf[cursor] == '}' {
**(**unsafe.Pointer)(unsafe.Pointer(&p)) = mapValue
cursor++
return cursor, nil
}
for {
k := unsafe_New(d.keyType)
keyCursor, err := d.keyDecoder.Decode(ctx, cursor, depth, k)
if err != nil {
return 0, err
}
cursor = skipWhiteSpace(buf, keyCursor)
if buf[cursor] != ':' {
return 0, errors.ErrExpected("colon after object key", cursor)
}
cursor++
v := unsafe_New(d.valueType)
valueCursor, err := d.valueDecoder.Decode(ctx, cursor, depth, v)
if err != nil {
return 0, err
}
d.mapassign(d.mapType, mapValue, k, v)
cursor = skipWhiteSpace(buf, valueCursor)
if buf[cursor] == '}' {
**(**unsafe.Pointer)(unsafe.Pointer(&p)) = mapValue
cursor++
return cursor, nil
}
if buf[cursor] != ',' {
return 0, errors.ErrExpected("comma after object value", cursor)
}
cursor++
}
}

View File

@@ -0,0 +1,108 @@
package decoder
import (
"encoding/json"
"strconv"
"unsafe"
"github.com/goccy/go-json/internal/errors"
)
type numberDecoder struct {
stringDecoder *stringDecoder
op func(unsafe.Pointer, json.Number)
structName string
fieldName string
}
func newNumberDecoder(structName, fieldName string, op func(unsafe.Pointer, json.Number)) *numberDecoder {
return &numberDecoder{
stringDecoder: newStringDecoder(structName, fieldName),
op: op,
structName: structName,
fieldName: fieldName,
}
}
func (d *numberDecoder) DecodeStream(s *Stream, depth int64, p unsafe.Pointer) error {
bytes, err := d.decodeStreamByte(s)
if err != nil {
return err
}
if _, err := strconv.ParseFloat(*(*string)(unsafe.Pointer(&bytes)), 64); err != nil {
return errors.ErrSyntax(err.Error(), s.totalOffset())
}
d.op(p, json.Number(string(bytes)))
s.reset()
return nil
}
func (d *numberDecoder) Decode(ctx *RuntimeContext, cursor, depth int64, p unsafe.Pointer) (int64, error) {
bytes, c, err := d.decodeByte(ctx.Buf, cursor)
if err != nil {
return 0, err
}
if _, err := strconv.ParseFloat(*(*string)(unsafe.Pointer(&bytes)), 64); err != nil {
return 0, errors.ErrSyntax(err.Error(), c)
}
cursor = c
s := *(*string)(unsafe.Pointer(&bytes))
d.op(p, json.Number(s))
return cursor, nil
}
func (d *numberDecoder) decodeStreamByte(s *Stream) ([]byte, error) {
for {
switch s.char() {
case ' ', '\n', '\t', '\r':
s.cursor++
continue
case '-', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9':
return floatBytes(s), nil
case 'n':
if err := nullBytes(s); err != nil {
return nil, err
}
return nil, nil
case '"':
return d.stringDecoder.decodeStreamByte(s)
case nul:
if s.read() {
continue
}
goto ERROR
default:
goto ERROR
}
}
ERROR:
return nil, errors.ErrUnexpectedEndOfJSON("json.Number", s.totalOffset())
}
func (d *numberDecoder) decodeByte(buf []byte, cursor int64) ([]byte, int64, error) {
for {
switch buf[cursor] {
case ' ', '\n', '\t', '\r':
cursor++
continue
case '-', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9':
start := cursor
cursor++
for floatTable[buf[cursor]] {
cursor++
}
num := buf[start:cursor]
return num, cursor, nil
case 'n':
if err := validateNull(buf, cursor); err != nil {
return nil, 0, err
}
cursor += 4
return nil, cursor, nil
case '"':
return d.stringDecoder.decodeByte(buf, cursor)
default:
return nil, 0, errors.ErrUnexpectedEndOfJSON("json.Number", cursor)
}
}
}

View File

@@ -0,0 +1,15 @@
package decoder
import "context"
type OptionFlags uint8
const (
FirstWinOption OptionFlags = 1 << iota
ContextOption
)
type Option struct {
Flags OptionFlags
Context context.Context
}

View File

@@ -0,0 +1,87 @@
package decoder
import (
"unsafe"
"github.com/goccy/go-json/internal/runtime"
)
type ptrDecoder struct {
dec Decoder
typ *runtime.Type
structName string
fieldName string
}
func newPtrDecoder(dec Decoder, typ *runtime.Type, structName, fieldName string) *ptrDecoder {
return &ptrDecoder{
dec: dec,
typ: typ,
structName: structName,
fieldName: fieldName,
}
}
func (d *ptrDecoder) contentDecoder() Decoder {
dec, ok := d.dec.(*ptrDecoder)
if !ok {
return d.dec
}
return dec.contentDecoder()
}
//nolint:golint
//go:linkname unsafe_New reflect.unsafe_New
func unsafe_New(*runtime.Type) unsafe.Pointer
func (d *ptrDecoder) DecodeStream(s *Stream, depth int64, p unsafe.Pointer) error {
if s.skipWhiteSpace() == nul {
s.read()
}
if s.char() == 'n' {
if err := nullBytes(s); err != nil {
return err
}
*(*unsafe.Pointer)(p) = nil
return nil
}
var newptr unsafe.Pointer
if *(*unsafe.Pointer)(p) == nil {
newptr = unsafe_New(d.typ)
*(*unsafe.Pointer)(p) = newptr
} else {
newptr = *(*unsafe.Pointer)(p)
}
if err := d.dec.DecodeStream(s, depth, newptr); err != nil {
return err
}
return nil
}
func (d *ptrDecoder) Decode(ctx *RuntimeContext, cursor, depth int64, p unsafe.Pointer) (int64, error) {
buf := ctx.Buf
cursor = skipWhiteSpace(buf, cursor)
if buf[cursor] == 'n' {
if err := validateNull(buf, cursor); err != nil {
return 0, err
}
if p != nil {
*(*unsafe.Pointer)(p) = nil
}
cursor += 4
return cursor, nil
}
var newptr unsafe.Pointer
if *(*unsafe.Pointer)(p) == nil {
newptr = unsafe_New(d.typ)
*(*unsafe.Pointer)(p) = newptr
} else {
newptr = *(*unsafe.Pointer)(p)
}
c, err := d.dec.Decode(ctx, cursor, depth, newptr)
if err != nil {
return 0, err
}
cursor = c
return cursor, nil
}

View File

@@ -0,0 +1,294 @@
package decoder
import (
"reflect"
"sync"
"unsafe"
"github.com/goccy/go-json/internal/errors"
"github.com/goccy/go-json/internal/runtime"
)
type sliceDecoder struct {
elemType *runtime.Type
isElemPointerType bool
valueDecoder Decoder
size uintptr
arrayPool sync.Pool
structName string
fieldName string
}
// If use reflect.SliceHeader, data type is uintptr.
// In this case, Go compiler cannot trace reference created by newArray().
// So, define using unsafe.Pointer as data type
type sliceHeader struct {
data unsafe.Pointer
len int
cap int
}
const (
defaultSliceCapacity = 2
)
func newSliceDecoder(dec Decoder, elemType *runtime.Type, size uintptr, structName, fieldName string) *sliceDecoder {
return &sliceDecoder{
valueDecoder: dec,
elemType: elemType,
isElemPointerType: elemType.Kind() == reflect.Ptr || elemType.Kind() == reflect.Map,
size: size,
arrayPool: sync.Pool{
New: func() interface{} {
return &sliceHeader{
data: newArray(elemType, defaultSliceCapacity),
len: 0,
cap: defaultSliceCapacity,
}
},
},
structName: structName,
fieldName: fieldName,
}
}
func (d *sliceDecoder) newSlice(src *sliceHeader) *sliceHeader {
slice := d.arrayPool.Get().(*sliceHeader)
if src.len > 0 {
// copy original elem
if slice.cap < src.cap {
data := newArray(d.elemType, src.cap)
slice = &sliceHeader{data: data, len: src.len, cap: src.cap}
} else {
slice.len = src.len
}
copySlice(d.elemType, *slice, *src)
} else {
slice.len = 0
}
return slice
}
func (d *sliceDecoder) releaseSlice(p *sliceHeader) {
d.arrayPool.Put(p)
}
//go:linkname copySlice reflect.typedslicecopy
func copySlice(elemType *runtime.Type, dst, src sliceHeader) int
//go:linkname newArray reflect.unsafe_NewArray
func newArray(*runtime.Type, int) unsafe.Pointer
//go:linkname typedmemmove reflect.typedmemmove
func typedmemmove(t *runtime.Type, dst, src unsafe.Pointer)
func (d *sliceDecoder) errNumber(offset int64) *errors.UnmarshalTypeError {
return &errors.UnmarshalTypeError{
Value: "number",
Type: reflect.SliceOf(runtime.RType2Type(d.elemType)),
Struct: d.structName,
Field: d.fieldName,
Offset: offset,
}
}
func (d *sliceDecoder) DecodeStream(s *Stream, depth int64, p unsafe.Pointer) error {
depth++
if depth > maxDecodeNestingDepth {
return errors.ErrExceededMaxDepth(s.char(), s.cursor)
}
for {
switch s.char() {
case ' ', '\n', '\t', '\r':
s.cursor++
continue
case 'n':
if err := nullBytes(s); err != nil {
return err
}
*(*unsafe.Pointer)(p) = nil
return nil
case '[':
s.cursor++
if s.skipWhiteSpace() == ']' {
dst := (*sliceHeader)(p)
if dst.data == nil {
dst.data = newArray(d.elemType, 0)
} else {
dst.len = 0
}
s.cursor++
return nil
}
idx := 0
slice := d.newSlice((*sliceHeader)(p))
srcLen := slice.len
capacity := slice.cap
data := slice.data
for {
if capacity <= idx {
src := sliceHeader{data: data, len: idx, cap: capacity}
capacity *= 2
data = newArray(d.elemType, capacity)
dst := sliceHeader{data: data, len: idx, cap: capacity}
copySlice(d.elemType, dst, src)
}
ep := unsafe.Pointer(uintptr(data) + uintptr(idx)*d.size)
// if srcLen is greater than idx, keep the original reference
if srcLen <= idx {
if d.isElemPointerType {
**(**unsafe.Pointer)(unsafe.Pointer(&ep)) = nil // initialize elem pointer
} else {
// assign new element to the slice
typedmemmove(d.elemType, ep, unsafe_New(d.elemType))
}
}
if err := d.valueDecoder.DecodeStream(s, depth, ep); err != nil {
return err
}
s.skipWhiteSpace()
RETRY:
switch s.char() {
case ']':
slice.cap = capacity
slice.len = idx + 1
slice.data = data
dst := (*sliceHeader)(p)
dst.len = idx + 1
if dst.len > dst.cap {
dst.data = newArray(d.elemType, dst.len)
dst.cap = dst.len
}
copySlice(d.elemType, *dst, *slice)
d.releaseSlice(slice)
s.cursor++
return nil
case ',':
idx++
case nul:
if s.read() {
goto RETRY
}
slice.cap = capacity
slice.data = data
d.releaseSlice(slice)
goto ERROR
default:
slice.cap = capacity
slice.data = data
d.releaseSlice(slice)
goto ERROR
}
s.cursor++
}
case '-', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9':
return d.errNumber(s.totalOffset())
case nul:
if s.read() {
continue
}
goto ERROR
default:
goto ERROR
}
}
ERROR:
return errors.ErrUnexpectedEndOfJSON("slice", s.totalOffset())
}
func (d *sliceDecoder) Decode(ctx *RuntimeContext, cursor, depth int64, p unsafe.Pointer) (int64, error) {
buf := ctx.Buf
depth++
if depth > maxDecodeNestingDepth {
return 0, errors.ErrExceededMaxDepth(buf[cursor], cursor)
}
for {
switch buf[cursor] {
case ' ', '\n', '\t', '\r':
cursor++
continue
case 'n':
if err := validateNull(buf, cursor); err != nil {
return 0, err
}
cursor += 4
*(*unsafe.Pointer)(p) = nil
return cursor, nil
case '[':
cursor++
cursor = skipWhiteSpace(buf, cursor)
if buf[cursor] == ']' {
dst := (*sliceHeader)(p)
if dst.data == nil {
dst.data = newArray(d.elemType, 0)
} else {
dst.len = 0
}
cursor++
return cursor, nil
}
idx := 0
slice := d.newSlice((*sliceHeader)(p))
srcLen := slice.len
capacity := slice.cap
data := slice.data
for {
if capacity <= idx {
src := sliceHeader{data: data, len: idx, cap: capacity}
capacity *= 2
data = newArray(d.elemType, capacity)
dst := sliceHeader{data: data, len: idx, cap: capacity}
copySlice(d.elemType, dst, src)
}
ep := unsafe.Pointer(uintptr(data) + uintptr(idx)*d.size)
// if srcLen is greater than idx, keep the original reference
if srcLen <= idx {
if d.isElemPointerType {
**(**unsafe.Pointer)(unsafe.Pointer(&ep)) = nil // initialize elem pointer
} else {
// assign new element to the slice
typedmemmove(d.elemType, ep, unsafe_New(d.elemType))
}
}
c, err := d.valueDecoder.Decode(ctx, cursor, depth, ep)
if err != nil {
return 0, err
}
cursor = c
cursor = skipWhiteSpace(buf, cursor)
switch buf[cursor] {
case ']':
slice.cap = capacity
slice.len = idx + 1
slice.data = data
dst := (*sliceHeader)(p)
dst.len = idx + 1
if dst.len > dst.cap {
dst.data = newArray(d.elemType, dst.len)
dst.cap = dst.len
}
copySlice(d.elemType, *dst, *slice)
d.releaseSlice(slice)
cursor++
return cursor, nil
case ',':
idx++
default:
slice.cap = capacity
slice.data = data
d.releaseSlice(slice)
return 0, errors.ErrInvalidCharacter(buf[cursor], "slice", cursor)
}
cursor++
}
case '-', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9':
return 0, d.errNumber(cursor)
default:
return 0, errors.ErrUnexpectedEndOfJSON("slice", cursor)
}
}
}

View File

@@ -0,0 +1,554 @@
package decoder
import (
"bytes"
"encoding/json"
"io"
"strconv"
"unsafe"
"github.com/goccy/go-json/internal/errors"
)
const (
initBufSize = 512
)
type Stream struct {
buf []byte
bufSize int64
length int64
r io.Reader
offset int64
cursor int64
filledBuffer bool
allRead bool
UseNumber bool
DisallowUnknownFields bool
Option *Option
}
func NewStream(r io.Reader) *Stream {
return &Stream{
r: r,
bufSize: initBufSize,
buf: make([]byte, initBufSize),
Option: &Option{},
}
}
func (s *Stream) TotalOffset() int64 {
return s.totalOffset()
}
func (s *Stream) Buffered() io.Reader {
buflen := int64(len(s.buf))
for i := s.cursor; i < buflen; i++ {
if s.buf[i] == nul {
return bytes.NewReader(s.buf[s.cursor:i])
}
}
return bytes.NewReader(s.buf[s.cursor:])
}
func (s *Stream) PrepareForDecode() error {
for {
switch s.char() {
case ' ', '\t', '\r', '\n':
s.cursor++
continue
case ',', ':':
s.cursor++
return nil
case nul:
if s.read() {
continue
}
return io.EOF
}
break
}
return nil
}
func (s *Stream) totalOffset() int64 {
return s.offset + s.cursor
}
func (s *Stream) char() byte {
return s.buf[s.cursor]
}
func (s *Stream) equalChar(c byte) bool {
cur := s.buf[s.cursor]
if cur == nul {
s.read()
cur = s.buf[s.cursor]
}
return cur == c
}
func (s *Stream) stat() ([]byte, int64, unsafe.Pointer) {
return s.buf, s.cursor, (*sliceHeader)(unsafe.Pointer(&s.buf)).data
}
func (s *Stream) bufptr() unsafe.Pointer {
return (*sliceHeader)(unsafe.Pointer(&s.buf)).data
}
func (s *Stream) statForRetry() ([]byte, int64, unsafe.Pointer) {
s.cursor-- // for retry ( because caller progress cursor position in each loop )
return s.buf, s.cursor, (*sliceHeader)(unsafe.Pointer(&s.buf)).data
}
func (s *Stream) Reset() {
s.reset()
s.bufSize = initBufSize
}
func (s *Stream) More() bool {
for {
switch s.char() {
case ' ', '\n', '\r', '\t':
s.cursor++
continue
case '}', ']':
return false
case nul:
if s.read() {
continue
}
return false
}
break
}
return true
}
func (s *Stream) Token() (interface{}, error) {
for {
c := s.char()
switch c {
case ' ', '\n', '\r', '\t':
s.cursor++
case '{', '[', ']', '}':
s.cursor++
return json.Delim(c), nil
case ',', ':':
s.cursor++
case '-', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9':
bytes := floatBytes(s)
s := *(*string)(unsafe.Pointer(&bytes))
f64, err := strconv.ParseFloat(s, 64)
if err != nil {
return nil, err
}
return f64, nil
case '"':
bytes, err := stringBytes(s)
if err != nil {
return nil, err
}
return string(bytes), nil
case 't':
if err := trueBytes(s); err != nil {
return nil, err
}
return true, nil
case 'f':
if err := falseBytes(s); err != nil {
return nil, err
}
return false, nil
case 'n':
if err := nullBytes(s); err != nil {
return nil, err
}
return nil, nil
case nul:
if s.read() {
continue
}
goto END
default:
return nil, errors.ErrInvalidCharacter(s.char(), "token", s.totalOffset())
}
}
END:
return nil, io.EOF
}
func (s *Stream) reset() {
s.offset += s.cursor
s.buf = s.buf[s.cursor:]
s.length -= s.cursor
s.cursor = 0
}
func (s *Stream) readBuf() []byte {
if s.filledBuffer {
s.bufSize *= 2
remainBuf := s.buf
s.buf = make([]byte, s.bufSize)
copy(s.buf, remainBuf)
}
remainLen := s.length - s.cursor
remainNotNulCharNum := int64(0)
for i := int64(0); i < remainLen; i++ {
if s.buf[s.cursor+i] == nul {
break
}
remainNotNulCharNum++
}
s.length = s.cursor + remainNotNulCharNum
return s.buf[s.cursor+remainNotNulCharNum:]
}
func (s *Stream) read() bool {
if s.allRead {
return false
}
buf := s.readBuf()
last := len(buf) - 1
buf[last] = nul
n, err := s.r.Read(buf[:last])
s.length += int64(n)
if n == last {
s.filledBuffer = true
} else {
s.filledBuffer = false
}
if err == io.EOF {
s.allRead = true
} else if err != nil {
return false
}
return true
}
func (s *Stream) skipWhiteSpace() byte {
p := s.bufptr()
LOOP:
c := char(p, s.cursor)
switch c {
case ' ', '\n', '\t', '\r':
s.cursor++
goto LOOP
case nul:
if s.read() {
p = s.bufptr()
goto LOOP
}
}
return c
}
func (s *Stream) skipObject(depth int64) error {
braceCount := 1
_, cursor, p := s.stat()
for {
switch char(p, cursor) {
case '{':
braceCount++
depth++
if depth > maxDecodeNestingDepth {
return errors.ErrExceededMaxDepth(s.char(), s.cursor)
}
case '}':
braceCount--
depth--
if braceCount == 0 {
s.cursor = cursor + 1
return nil
}
case '[':
depth++
if depth > maxDecodeNestingDepth {
return errors.ErrExceededMaxDepth(s.char(), s.cursor)
}
case ']':
depth--
case '"':
for {
cursor++
switch char(p, cursor) {
case '\\':
cursor++
if char(p, cursor) == nul {
s.cursor = cursor
if s.read() {
_, cursor, p = s.statForRetry()
continue
}
return errors.ErrUnexpectedEndOfJSON("string of object", cursor)
}
case '"':
goto SWITCH_OUT
case nul:
s.cursor = cursor
if s.read() {
_, cursor, p = s.statForRetry()
continue
}
return errors.ErrUnexpectedEndOfJSON("string of object", cursor)
}
}
case nul:
s.cursor = cursor
if s.read() {
_, cursor, p = s.stat()
continue
}
return errors.ErrUnexpectedEndOfJSON("object of object", cursor)
}
SWITCH_OUT:
cursor++
}
}
func (s *Stream) skipArray(depth int64) error {
bracketCount := 1
_, cursor, p := s.stat()
for {
switch char(p, cursor) {
case '[':
bracketCount++
depth++
if depth > maxDecodeNestingDepth {
return errors.ErrExceededMaxDepth(s.char(), s.cursor)
}
case ']':
bracketCount--
depth--
if bracketCount == 0 {
s.cursor = cursor + 1
return nil
}
case '{':
depth++
if depth > maxDecodeNestingDepth {
return errors.ErrExceededMaxDepth(s.char(), s.cursor)
}
case '}':
depth--
case '"':
for {
cursor++
switch char(p, cursor) {
case '\\':
cursor++
if char(p, cursor) == nul {
s.cursor = cursor
if s.read() {
_, cursor, p = s.statForRetry()
continue
}
return errors.ErrUnexpectedEndOfJSON("string of object", cursor)
}
case '"':
goto SWITCH_OUT
case nul:
s.cursor = cursor
if s.read() {
_, cursor, p = s.statForRetry()
continue
}
return errors.ErrUnexpectedEndOfJSON("string of object", cursor)
}
}
case nul:
s.cursor = cursor
if s.read() {
_, cursor, p = s.stat()
continue
}
return errors.ErrUnexpectedEndOfJSON("array of object", cursor)
}
SWITCH_OUT:
cursor++
}
}
func (s *Stream) skipValue(depth int64) error {
_, cursor, p := s.stat()
for {
switch char(p, cursor) {
case ' ', '\n', '\t', '\r':
cursor++
continue
case nul:
s.cursor = cursor
if s.read() {
_, cursor, p = s.stat()
continue
}
return errors.ErrUnexpectedEndOfJSON("value of object", s.totalOffset())
case '{':
s.cursor = cursor + 1
return s.skipObject(depth + 1)
case '[':
s.cursor = cursor + 1
return s.skipArray(depth + 1)
case '"':
for {
cursor++
switch char(p, cursor) {
case '\\':
cursor++
if char(p, cursor) == nul {
s.cursor = cursor
if s.read() {
_, cursor, p = s.statForRetry()
continue
}
return errors.ErrUnexpectedEndOfJSON("value of string", s.totalOffset())
}
case '"':
s.cursor = cursor + 1
return nil
case nul:
s.cursor = cursor
if s.read() {
_, cursor, p = s.statForRetry()
continue
}
return errors.ErrUnexpectedEndOfJSON("value of string", s.totalOffset())
}
}
case '-', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9':
for {
cursor++
c := char(p, cursor)
if floatTable[c] {
continue
} else if c == nul {
if s.read() {
s.cursor-- // for retry current character
_, cursor, p = s.stat()
continue
}
}
s.cursor = cursor
return nil
}
case 't':
s.cursor = cursor
if err := trueBytes(s); err != nil {
return err
}
return nil
case 'f':
s.cursor = cursor
if err := falseBytes(s); err != nil {
return err
}
return nil
case 'n':
s.cursor = cursor
if err := nullBytes(s); err != nil {
return err
}
return nil
}
cursor++
}
}
func nullBytes(s *Stream) error {
// current cursor's character is 'n'
s.cursor++
if s.char() != 'u' {
if err := retryReadNull(s); err != nil {
return err
}
}
s.cursor++
if s.char() != 'l' {
if err := retryReadNull(s); err != nil {
return err
}
}
s.cursor++
if s.char() != 'l' {
if err := retryReadNull(s); err != nil {
return err
}
}
s.cursor++
return nil
}
func retryReadNull(s *Stream) error {
if s.char() == nul && s.read() {
return nil
}
return errors.ErrInvalidCharacter(s.char(), "null", s.totalOffset())
}
func trueBytes(s *Stream) error {
// current cursor's character is 't'
s.cursor++
if s.char() != 'r' {
if err := retryReadTrue(s); err != nil {
return err
}
}
s.cursor++
if s.char() != 'u' {
if err := retryReadTrue(s); err != nil {
return err
}
}
s.cursor++
if s.char() != 'e' {
if err := retryReadTrue(s); err != nil {
return err
}
}
s.cursor++
return nil
}
func retryReadTrue(s *Stream) error {
if s.char() == nul && s.read() {
return nil
}
return errors.ErrInvalidCharacter(s.char(), "bool(true)", s.totalOffset())
}
func falseBytes(s *Stream) error {
// current cursor's character is 'f'
s.cursor++
if s.char() != 'a' {
if err := retryReadFalse(s); err != nil {
return err
}
}
s.cursor++
if s.char() != 'l' {
if err := retryReadFalse(s); err != nil {
return err
}
}
s.cursor++
if s.char() != 's' {
if err := retryReadFalse(s); err != nil {
return err
}
}
s.cursor++
if s.char() != 'e' {
if err := retryReadFalse(s); err != nil {
return err
}
}
s.cursor++
return nil
}
func retryReadFalse(s *Stream) error {
if s.char() == nul && s.read() {
return nil
}
return errors.ErrInvalidCharacter(s.char(), "bool(false)", s.totalOffset())
}

View File

@@ -0,0 +1,361 @@
package decoder
import (
"reflect"
"unicode"
"unicode/utf16"
"unicode/utf8"
"unsafe"
"github.com/goccy/go-json/internal/errors"
)
type stringDecoder struct {
structName string
fieldName string
}
func newStringDecoder(structName, fieldName string) *stringDecoder {
return &stringDecoder{
structName: structName,
fieldName: fieldName,
}
}
func (d *stringDecoder) errUnmarshalType(typeName string, offset int64) *errors.UnmarshalTypeError {
return &errors.UnmarshalTypeError{
Value: typeName,
Type: reflect.TypeOf(""),
Offset: offset,
Struct: d.structName,
Field: d.fieldName,
}
}
func (d *stringDecoder) DecodeStream(s *Stream, depth int64, p unsafe.Pointer) error {
bytes, err := d.decodeStreamByte(s)
if err != nil {
return err
}
if bytes == nil {
return nil
}
**(**string)(unsafe.Pointer(&p)) = *(*string)(unsafe.Pointer(&bytes))
s.reset()
return nil
}
func (d *stringDecoder) Decode(ctx *RuntimeContext, cursor, depth int64, p unsafe.Pointer) (int64, error) {
bytes, c, err := d.decodeByte(ctx.Buf, cursor)
if err != nil {
return 0, err
}
if bytes == nil {
return c, nil
}
cursor = c
**(**string)(unsafe.Pointer(&p)) = *(*string)(unsafe.Pointer(&bytes))
return cursor, nil
}
var (
hexToInt = [256]int{
'0': 0,
'1': 1,
'2': 2,
'3': 3,
'4': 4,
'5': 5,
'6': 6,
'7': 7,
'8': 8,
'9': 9,
'A': 10,
'B': 11,
'C': 12,
'D': 13,
'E': 14,
'F': 15,
'a': 10,
'b': 11,
'c': 12,
'd': 13,
'e': 14,
'f': 15,
}
)
func unicodeToRune(code []byte) rune {
var r rune
for i := 0; i < len(code); i++ {
r = r*16 + rune(hexToInt[code[i]])
}
return r
}
func decodeUnicodeRune(s *Stream, p unsafe.Pointer) (rune, int64, unsafe.Pointer, error) {
const defaultOffset = 5
const surrogateOffset = 11
if s.cursor+defaultOffset >= s.length {
if !s.read() {
return rune(0), 0, nil, errors.ErrInvalidCharacter(s.char(), "escaped string", s.totalOffset())
}
p = s.bufptr()
}
r := unicodeToRune(s.buf[s.cursor+1 : s.cursor+defaultOffset])
if utf16.IsSurrogate(r) {
if s.cursor+surrogateOffset >= s.length {
s.read()
p = s.bufptr()
}
if s.cursor+surrogateOffset >= s.length || s.buf[s.cursor+defaultOffset] != '\\' || s.buf[s.cursor+defaultOffset+1] != 'u' {
return unicode.ReplacementChar, defaultOffset, p, nil
}
r2 := unicodeToRune(s.buf[s.cursor+defaultOffset+2 : s.cursor+surrogateOffset])
if r := utf16.DecodeRune(r, r2); r != unicode.ReplacementChar {
return r, surrogateOffset, p, nil
}
}
return r, defaultOffset, p, nil
}
func decodeUnicode(s *Stream, p unsafe.Pointer) (unsafe.Pointer, error) {
const backSlashAndULen = 2 // length of \u
r, offset, pp, err := decodeUnicodeRune(s, p)
if err != nil {
return nil, err
}
unicode := []byte(string(r))
unicodeLen := int64(len(unicode))
s.buf = append(append(s.buf[:s.cursor-1], unicode...), s.buf[s.cursor+offset:]...)
unicodeOrgLen := offset - 1
s.length = s.length - (backSlashAndULen + (unicodeOrgLen - unicodeLen))
s.cursor = s.cursor - backSlashAndULen + unicodeLen
return pp, nil
}
func decodeEscapeString(s *Stream, p unsafe.Pointer) (unsafe.Pointer, error) {
s.cursor++
RETRY:
switch s.buf[s.cursor] {
case '"':
s.buf[s.cursor] = '"'
case '\\':
s.buf[s.cursor] = '\\'
case '/':
s.buf[s.cursor] = '/'
case 'b':
s.buf[s.cursor] = '\b'
case 'f':
s.buf[s.cursor] = '\f'
case 'n':
s.buf[s.cursor] = '\n'
case 'r':
s.buf[s.cursor] = '\r'
case 't':
s.buf[s.cursor] = '\t'
case 'u':
return decodeUnicode(s, p)
case nul:
if !s.read() {
return nil, errors.ErrInvalidCharacter(s.char(), "escaped string", s.totalOffset())
}
goto RETRY
default:
return nil, errors.ErrUnexpectedEndOfJSON("string", s.totalOffset())
}
s.buf = append(s.buf[:s.cursor-1], s.buf[s.cursor:]...)
s.length--
s.cursor--
return p, nil
}
var (
runeErrBytes = []byte(string(utf8.RuneError))
runeErrBytesLen = int64(len(runeErrBytes))
)
func stringBytes(s *Stream) ([]byte, error) {
_, cursor, p := s.stat()
cursor++ // skip double quote char
start := cursor
for {
switch char(p, cursor) {
case '\\':
s.cursor = cursor
pp, err := decodeEscapeString(s, p)
if err != nil {
return nil, err
}
p = pp
cursor = s.cursor
case '"':
literal := s.buf[start:cursor]
cursor++
s.cursor = cursor
return literal, nil
case
// 0x00 is nul, 0x5c is '\\', 0x22 is '"' .
0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, // 0x00-0x0F
0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, 0x18, 0x19, 0x1A, 0x1B, 0x1C, 0x1D, 0x1E, 0x1F, // 0x10-0x1F
0x20, 0x21 /*0x22,*/, 0x23, 0x24, 0x25, 0x26, 0x27, 0x28, 0x29, 0x2A, 0x2B, 0x2C, 0x2D, 0x2E, 0x2F, // 0x20-0x2F
0x30, 0x31, 0x32, 0x33, 0x34, 0x35, 0x36, 0x37, 0x38, 0x39, 0x3A, 0x3B, 0x3C, 0x3D, 0x3E, 0x3F, // 0x30-0x3F
0x40, 0x41, 0x42, 0x43, 0x44, 0x45, 0x46, 0x47, 0x48, 0x49, 0x4A, 0x4B, 0x4C, 0x4D, 0x4E, 0x4F, // 0x40-0x4F
0x50, 0x51, 0x52, 0x53, 0x54, 0x55, 0x56, 0x57, 0x58, 0x59, 0x5A, 0x5B /*0x5C,*/, 0x5D, 0x5E, 0x5F, // 0x50-0x5F
0x60, 0x61, 0x62, 0x63, 0x64, 0x65, 0x66, 0x67, 0x68, 0x69, 0x6A, 0x6B, 0x6C, 0x6D, 0x6E, 0x6F, // 0x60-0x6F
0x70, 0x71, 0x72, 0x73, 0x74, 0x75, 0x76, 0x77, 0x78, 0x79, 0x7A, 0x7B, 0x7C, 0x7D, 0x7E, 0x7F: // 0x70-0x7F
// character is ASCII. skip to next char
case
0x80, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87, 0x88, 0x89, 0x8A, 0x8B, 0x8C, 0x8D, 0x8E, 0x8F, // 0x80-0x8F
0x90, 0x91, 0x92, 0x93, 0x94, 0x95, 0x96, 0x97, 0x98, 0x99, 0x9A, 0x9B, 0x9C, 0x9D, 0x9E, 0x9F, // 0x90-0x9F
0xA0, 0xA1, 0xA2, 0xA3, 0xA4, 0xA5, 0xA6, 0xA7, 0xA8, 0xA9, 0xAA, 0xAB, 0xAC, 0xAD, 0xAE, 0xAF, // 0xA0-0xAF
0xB0, 0xB1, 0xB2, 0xB3, 0xB4, 0xB5, 0xB6, 0xB7, 0xB8, 0xB9, 0xBA, 0xBB, 0xBC, 0xBD, 0xBE, 0xBF, // 0xB0-0xBF
0xC0, 0xC1, // 0xC0-0xC1
0xF5, 0xF6, 0xF7, 0xF8, 0xF9, 0xFA, 0xFB, 0xFC, 0xFD, 0xFE, 0xFF: // 0xF5-0xFE
// character is invalid
s.buf = append(append(append([]byte{}, s.buf[:cursor]...), runeErrBytes...), s.buf[cursor+1:]...)
_, _, p = s.stat()
cursor += runeErrBytesLen
s.length += runeErrBytesLen
continue
case nul:
s.cursor = cursor
if s.read() {
_, cursor, p = s.stat()
continue
}
goto ERROR
case 0xEF:
// RuneError is {0xEF, 0xBF, 0xBD}
if s.buf[cursor+1] == 0xBF && s.buf[cursor+2] == 0xBD {
// found RuneError: skip
cursor += 2
break
}
fallthrough
default:
// multi bytes character
r, _ := utf8.DecodeRune(s.buf[cursor:])
b := []byte(string(r))
if r == utf8.RuneError {
s.buf = append(append(append([]byte{}, s.buf[:cursor]...), b...), s.buf[cursor+1:]...)
_, _, p = s.stat()
}
cursor += int64(len(b))
s.length += int64(len(b))
continue
}
cursor++
}
ERROR:
return nil, errors.ErrUnexpectedEndOfJSON("string", s.totalOffset())
}
func (d *stringDecoder) decodeStreamByte(s *Stream) ([]byte, error) {
for {
switch s.char() {
case ' ', '\n', '\t', '\r':
s.cursor++
continue
case '[':
return nil, d.errUnmarshalType("array", s.totalOffset())
case '{':
return nil, d.errUnmarshalType("object", s.totalOffset())
case '-', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9':
return nil, d.errUnmarshalType("number", s.totalOffset())
case '"':
return stringBytes(s)
case 'n':
if err := nullBytes(s); err != nil {
return nil, err
}
return nil, nil
case nul:
if s.read() {
continue
}
}
break
}
return nil, errors.ErrNotAtBeginningOfValue(s.totalOffset())
}
func (d *stringDecoder) decodeByte(buf []byte, cursor int64) ([]byte, int64, error) {
for {
switch buf[cursor] {
case ' ', '\n', '\t', '\r':
cursor++
case '[':
return nil, 0, d.errUnmarshalType("array", cursor)
case '{':
return nil, 0, d.errUnmarshalType("object", cursor)
case '-', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9':
return nil, 0, d.errUnmarshalType("number", cursor)
case '"':
cursor++
start := cursor
b := (*sliceHeader)(unsafe.Pointer(&buf)).data
for {
switch char(b, cursor) {
case '\\':
cursor++
switch char(b, cursor) {
case '"':
buf[cursor] = '"'
buf = append(buf[:cursor-1], buf[cursor:]...)
case '\\':
buf[cursor] = '\\'
buf = append(buf[:cursor-1], buf[cursor:]...)
case '/':
buf[cursor] = '/'
buf = append(buf[:cursor-1], buf[cursor:]...)
case 'b':
buf[cursor] = '\b'
buf = append(buf[:cursor-1], buf[cursor:]...)
case 'f':
buf[cursor] = '\f'
buf = append(buf[:cursor-1], buf[cursor:]...)
case 'n':
buf[cursor] = '\n'
buf = append(buf[:cursor-1], buf[cursor:]...)
case 'r':
buf[cursor] = '\r'
buf = append(buf[:cursor-1], buf[cursor:]...)
case 't':
buf[cursor] = '\t'
buf = append(buf[:cursor-1], buf[cursor:]...)
case 'u':
buflen := int64(len(buf))
if cursor+5 >= buflen {
return nil, 0, errors.ErrUnexpectedEndOfJSON("escaped string", cursor)
}
code := unicodeToRune(buf[cursor+1 : cursor+5])
unicode := []byte(string(code))
buf = append(append(buf[:cursor-1], unicode...), buf[cursor+5:]...)
default:
return nil, 0, errors.ErrUnexpectedEndOfJSON("escaped string", cursor)
}
continue
case '"':
literal := buf[start:cursor]
cursor++
return literal, cursor, nil
case nul:
return nil, 0, errors.ErrUnexpectedEndOfJSON("string", cursor)
}
cursor++
}
case 'n':
if err := validateNull(buf, cursor); err != nil {
return nil, 0, err
}
cursor += 4
return nil, cursor, nil
default:
return nil, 0, errors.ErrNotAtBeginningOfValue(cursor)
}
}
}

View File

@@ -0,0 +1,819 @@
package decoder
import (
"fmt"
"math"
"math/bits"
"sort"
"strings"
"unicode"
"unicode/utf16"
"unsafe"
"github.com/goccy/go-json/internal/errors"
)
type structFieldSet struct {
dec Decoder
offset uintptr
isTaggedKey bool
fieldIdx int
key string
keyLen int64
err error
}
type structDecoder struct {
fieldMap map[string]*structFieldSet
fieldUniqueNameNum int
stringDecoder *stringDecoder
structName string
fieldName string
isTriedOptimize bool
keyBitmapUint8 [][256]uint8
keyBitmapUint16 [][256]uint16
sortedFieldSets []*structFieldSet
keyDecoder func(*structDecoder, []byte, int64) (int64, *structFieldSet, error)
keyStreamDecoder func(*structDecoder, *Stream) (*structFieldSet, string, error)
}
var (
largeToSmallTable [256]byte
)
func init() {
for i := 0; i < 256; i++ {
c := i
if 'A' <= c && c <= 'Z' {
c += 'a' - 'A'
}
largeToSmallTable[i] = byte(c)
}
}
func newStructDecoder(structName, fieldName string, fieldMap map[string]*structFieldSet) *structDecoder {
return &structDecoder{
fieldMap: fieldMap,
stringDecoder: newStringDecoder(structName, fieldName),
structName: structName,
fieldName: fieldName,
keyDecoder: decodeKey,
keyStreamDecoder: decodeKeyStream,
}
}
const (
allowOptimizeMaxKeyLen = 64
allowOptimizeMaxFieldLen = 16
)
func (d *structDecoder) tryOptimize() {
fieldUniqueNameMap := map[string]int{}
fieldIdx := -1
for k, v := range d.fieldMap {
lower := strings.ToLower(k)
idx, exists := fieldUniqueNameMap[lower]
if exists {
v.fieldIdx = idx
} else {
fieldIdx++
v.fieldIdx = fieldIdx
}
fieldUniqueNameMap[lower] = fieldIdx
}
d.fieldUniqueNameNum = len(fieldUniqueNameMap)
if d.isTriedOptimize {
return
}
fieldMap := map[string]*structFieldSet{}
conflicted := map[string]struct{}{}
for k, v := range d.fieldMap {
key := strings.ToLower(k)
if key != k {
// already exists same key (e.g. Hello and HELLO has same lower case key
if _, exists := conflicted[key]; exists {
d.isTriedOptimize = true
return
}
conflicted[key] = struct{}{}
}
if field, exists := fieldMap[key]; exists {
if field != v {
d.isTriedOptimize = true
return
}
}
fieldMap[key] = v
}
if len(fieldMap) > allowOptimizeMaxFieldLen {
d.isTriedOptimize = true
return
}
var maxKeyLen int
sortedKeys := []string{}
for key := range fieldMap {
keyLen := len(key)
if keyLen > allowOptimizeMaxKeyLen {
d.isTriedOptimize = true
return
}
if maxKeyLen < keyLen {
maxKeyLen = keyLen
}
sortedKeys = append(sortedKeys, key)
}
sort.Strings(sortedKeys)
// By allocating one extra capacity than `maxKeyLen`,
// it is possible to avoid the process of comparing the index of the key with the length of the bitmap each time.
bitmapLen := maxKeyLen + 1
if len(sortedKeys) <= 8 {
keyBitmap := make([][256]uint8, bitmapLen)
for i, key := range sortedKeys {
for j := 0; j < len(key); j++ {
c := key[j]
keyBitmap[j][c] |= (1 << uint(i))
}
d.sortedFieldSets = append(d.sortedFieldSets, fieldMap[key])
}
d.keyBitmapUint8 = keyBitmap
d.keyDecoder = decodeKeyByBitmapUint8
d.keyStreamDecoder = decodeKeyByBitmapUint8Stream
} else {
keyBitmap := make([][256]uint16, bitmapLen)
for i, key := range sortedKeys {
for j := 0; j < len(key); j++ {
c := key[j]
keyBitmap[j][c] |= (1 << uint(i))
}
d.sortedFieldSets = append(d.sortedFieldSets, fieldMap[key])
}
d.keyBitmapUint16 = keyBitmap
d.keyDecoder = decodeKeyByBitmapUint16
d.keyStreamDecoder = decodeKeyByBitmapUint16Stream
}
}
// decode from '\uXXXX'
func decodeKeyCharByUnicodeRune(buf []byte, cursor int64) ([]byte, int64) {
const defaultOffset = 4
const surrogateOffset = 6
r := unicodeToRune(buf[cursor : cursor+defaultOffset])
if utf16.IsSurrogate(r) {
cursor += defaultOffset
if cursor+surrogateOffset >= int64(len(buf)) || buf[cursor] != '\\' || buf[cursor+1] != 'u' {
return []byte(string(unicode.ReplacementChar)), cursor + defaultOffset - 1
}
cursor += 2
r2 := unicodeToRune(buf[cursor : cursor+defaultOffset])
if r := utf16.DecodeRune(r, r2); r != unicode.ReplacementChar {
return []byte(string(r)), cursor + defaultOffset - 1
}
}
return []byte(string(r)), cursor + defaultOffset - 1
}
func decodeKeyCharByEscapedChar(buf []byte, cursor int64) ([]byte, int64) {
c := buf[cursor]
cursor++
switch c {
case '"':
return []byte{'"'}, cursor
case '\\':
return []byte{'\\'}, cursor
case '/':
return []byte{'/'}, cursor
case 'b':
return []byte{'\b'}, cursor
case 'f':
return []byte{'\f'}, cursor
case 'n':
return []byte{'\n'}, cursor
case 'r':
return []byte{'\r'}, cursor
case 't':
return []byte{'\t'}, cursor
case 'u':
return decodeKeyCharByUnicodeRune(buf, cursor)
}
return nil, cursor
}
func decodeKeyByBitmapUint8(d *structDecoder, buf []byte, cursor int64) (int64, *structFieldSet, error) {
var (
curBit uint8 = math.MaxUint8
)
b := (*sliceHeader)(unsafe.Pointer(&buf)).data
for {
switch char(b, cursor) {
case ' ', '\n', '\t', '\r':
cursor++
case '"':
cursor++
c := char(b, cursor)
switch c {
case '"':
cursor++
return cursor, nil, nil
case nul:
return 0, nil, errors.ErrUnexpectedEndOfJSON("string", cursor)
}
keyIdx := 0
bitmap := d.keyBitmapUint8
start := cursor
for {
c := char(b, cursor)
switch c {
case '"':
fieldSetIndex := bits.TrailingZeros8(curBit)
field := d.sortedFieldSets[fieldSetIndex]
keyLen := cursor - start
cursor++
if keyLen < field.keyLen {
// early match
return cursor, nil, nil
}
return cursor, field, nil
case nul:
return 0, nil, errors.ErrUnexpectedEndOfJSON("string", cursor)
case '\\':
cursor++
chars, nextCursor := decodeKeyCharByEscapedChar(buf, cursor)
for _, c := range chars {
curBit &= bitmap[keyIdx][largeToSmallTable[c]]
if curBit == 0 {
return decodeKeyNotFound(b, cursor)
}
keyIdx++
}
cursor = nextCursor
default:
curBit &= bitmap[keyIdx][largeToSmallTable[c]]
if curBit == 0 {
return decodeKeyNotFound(b, cursor)
}
keyIdx++
}
cursor++
}
default:
return cursor, nil, errors.ErrNotAtBeginningOfValue(cursor)
}
}
}
func decodeKeyByBitmapUint16(d *structDecoder, buf []byte, cursor int64) (int64, *structFieldSet, error) {
var (
curBit uint16 = math.MaxUint16
)
b := (*sliceHeader)(unsafe.Pointer(&buf)).data
for {
switch char(b, cursor) {
case ' ', '\n', '\t', '\r':
cursor++
case '"':
cursor++
c := char(b, cursor)
switch c {
case '"':
cursor++
return cursor, nil, nil
case nul:
return 0, nil, errors.ErrUnexpectedEndOfJSON("string", cursor)
}
keyIdx := 0
bitmap := d.keyBitmapUint16
start := cursor
for {
c := char(b, cursor)
switch c {
case '"':
fieldSetIndex := bits.TrailingZeros16(curBit)
field := d.sortedFieldSets[fieldSetIndex]
keyLen := cursor - start
cursor++
if keyLen < field.keyLen {
// early match
return cursor, nil, nil
}
return cursor, field, nil
case nul:
return 0, nil, errors.ErrUnexpectedEndOfJSON("string", cursor)
case '\\':
cursor++
chars, nextCursor := decodeKeyCharByEscapedChar(buf, cursor)
for _, c := range chars {
curBit &= bitmap[keyIdx][largeToSmallTable[c]]
if curBit == 0 {
return decodeKeyNotFound(b, cursor)
}
keyIdx++
}
cursor = nextCursor
default:
curBit &= bitmap[keyIdx][largeToSmallTable[c]]
if curBit == 0 {
return decodeKeyNotFound(b, cursor)
}
keyIdx++
}
cursor++
}
default:
return cursor, nil, errors.ErrNotAtBeginningOfValue(cursor)
}
}
}
func decodeKeyNotFound(b unsafe.Pointer, cursor int64) (int64, *structFieldSet, error) {
for {
cursor++
switch char(b, cursor) {
case '"':
cursor++
return cursor, nil, nil
case '\\':
cursor++
if char(b, cursor) == nul {
return 0, nil, errors.ErrUnexpectedEndOfJSON("string", cursor)
}
case nul:
return 0, nil, errors.ErrUnexpectedEndOfJSON("string", cursor)
}
}
}
func decodeKey(d *structDecoder, buf []byte, cursor int64) (int64, *structFieldSet, error) {
key, c, err := d.stringDecoder.decodeByte(buf, cursor)
if err != nil {
return 0, nil, err
}
cursor = c
k := *(*string)(unsafe.Pointer(&key))
field, exists := d.fieldMap[k]
if !exists {
return cursor, nil, nil
}
return cursor, field, nil
}
func decodeKeyByBitmapUint8Stream(d *structDecoder, s *Stream) (*structFieldSet, string, error) {
var (
curBit uint8 = math.MaxUint8
)
_, cursor, p := s.stat()
for {
switch char(p, cursor) {
case ' ', '\n', '\t', '\r':
cursor++
case nul:
s.cursor = cursor
if s.read() {
_, cursor, p = s.stat()
continue
}
return nil, "", errors.ErrNotAtBeginningOfValue(s.totalOffset())
case '"':
cursor++
FIRST_CHAR:
start := cursor
switch char(p, cursor) {
case '"':
cursor++
s.cursor = cursor
return nil, "", nil
case nul:
s.cursor = cursor
if s.read() {
_, cursor, p = s.stat()
goto FIRST_CHAR
}
return nil, "", errors.ErrUnexpectedEndOfJSON("string", s.totalOffset())
}
keyIdx := 0
bitmap := d.keyBitmapUint8
for {
c := char(p, cursor)
switch c {
case '"':
fieldSetIndex := bits.TrailingZeros8(curBit)
field := d.sortedFieldSets[fieldSetIndex]
keyLen := cursor - start
cursor++
s.cursor = cursor
if keyLen < field.keyLen {
// early match
return nil, field.key, nil
}
return field, field.key, nil
case nul:
s.cursor = cursor
if s.read() {
_, cursor, p = s.stat()
continue
}
return nil, "", errors.ErrUnexpectedEndOfJSON("string", s.totalOffset())
case '\\':
s.cursor = cursor + 1 // skip '\' char
chars, err := decodeKeyCharByEscapeCharStream(s)
if err != nil {
return nil, "", err
}
cursor = s.cursor
for _, c := range chars {
curBit &= bitmap[keyIdx][largeToSmallTable[c]]
if curBit == 0 {
s.cursor = cursor
return decodeKeyNotFoundStream(s, start)
}
keyIdx++
}
default:
curBit &= bitmap[keyIdx][largeToSmallTable[c]]
if curBit == 0 {
s.cursor = cursor
return decodeKeyNotFoundStream(s, start)
}
keyIdx++
}
cursor++
}
default:
return nil, "", errors.ErrNotAtBeginningOfValue(s.totalOffset())
}
}
}
func decodeKeyByBitmapUint16Stream(d *structDecoder, s *Stream) (*structFieldSet, string, error) {
var (
curBit uint16 = math.MaxUint16
)
_, cursor, p := s.stat()
for {
switch char(p, cursor) {
case ' ', '\n', '\t', '\r':
cursor++
case nul:
s.cursor = cursor
if s.read() {
_, cursor, p = s.stat()
continue
}
return nil, "", errors.ErrNotAtBeginningOfValue(s.totalOffset())
case '"':
cursor++
FIRST_CHAR:
start := cursor
switch char(p, cursor) {
case '"':
cursor++
s.cursor = cursor
return nil, "", nil
case nul:
s.cursor = cursor
if s.read() {
_, cursor, p = s.stat()
goto FIRST_CHAR
}
return nil, "", errors.ErrUnexpectedEndOfJSON("string", s.totalOffset())
}
keyIdx := 0
bitmap := d.keyBitmapUint16
for {
c := char(p, cursor)
switch c {
case '"':
fieldSetIndex := bits.TrailingZeros16(curBit)
field := d.sortedFieldSets[fieldSetIndex]
keyLen := cursor - start
cursor++
s.cursor = cursor
if keyLen < field.keyLen {
// early match
return nil, field.key, nil
}
return field, field.key, nil
case nul:
s.cursor = cursor
if s.read() {
_, cursor, p = s.stat()
continue
}
return nil, "", errors.ErrUnexpectedEndOfJSON("string", s.totalOffset())
case '\\':
s.cursor = cursor + 1 // skip '\' char
chars, err := decodeKeyCharByEscapeCharStream(s)
if err != nil {
return nil, "", err
}
cursor = s.cursor
for _, c := range chars {
curBit &= bitmap[keyIdx][largeToSmallTable[c]]
if curBit == 0 {
s.cursor = cursor
return decodeKeyNotFoundStream(s, start)
}
keyIdx++
}
default:
curBit &= bitmap[keyIdx][largeToSmallTable[c]]
if curBit == 0 {
s.cursor = cursor
return decodeKeyNotFoundStream(s, start)
}
keyIdx++
}
cursor++
}
default:
return nil, "", errors.ErrNotAtBeginningOfValue(s.totalOffset())
}
}
}
// decode from '\uXXXX'
func decodeKeyCharByUnicodeRuneStream(s *Stream) ([]byte, error) {
const defaultOffset = 4
const surrogateOffset = 6
if s.cursor+defaultOffset >= s.length {
if !s.read() {
return nil, errors.ErrInvalidCharacter(s.char(), "escaped unicode char", s.totalOffset())
}
}
r := unicodeToRune(s.buf[s.cursor : s.cursor+defaultOffset])
if utf16.IsSurrogate(r) {
s.cursor += defaultOffset
if s.cursor+surrogateOffset >= s.length {
s.read()
}
if s.cursor+surrogateOffset >= s.length || s.buf[s.cursor] != '\\' || s.buf[s.cursor+1] != 'u' {
s.cursor += defaultOffset - 1
return []byte(string(unicode.ReplacementChar)), nil
}
r2 := unicodeToRune(s.buf[s.cursor+defaultOffset+2 : s.cursor+surrogateOffset])
if r := utf16.DecodeRune(r, r2); r != unicode.ReplacementChar {
s.cursor += defaultOffset - 1
return []byte(string(r)), nil
}
}
s.cursor += defaultOffset - 1
return []byte(string(r)), nil
}
func decodeKeyCharByEscapeCharStream(s *Stream) ([]byte, error) {
c := s.buf[s.cursor]
s.cursor++
RETRY:
switch c {
case '"':
return []byte{'"'}, nil
case '\\':
return []byte{'\\'}, nil
case '/':
return []byte{'/'}, nil
case 'b':
return []byte{'\b'}, nil
case 'f':
return []byte{'\f'}, nil
case 'n':
return []byte{'\n'}, nil
case 'r':
return []byte{'\r'}, nil
case 't':
return []byte{'\t'}, nil
case 'u':
return decodeKeyCharByUnicodeRuneStream(s)
case nul:
if !s.read() {
return nil, errors.ErrInvalidCharacter(s.char(), "escaped char", s.totalOffset())
}
goto RETRY
default:
return nil, errors.ErrUnexpectedEndOfJSON("struct field", s.totalOffset())
}
}
func decodeKeyNotFoundStream(s *Stream, start int64) (*structFieldSet, string, error) {
buf, cursor, p := s.stat()
for {
cursor++
switch char(p, cursor) {
case '"':
b := buf[start:cursor]
key := *(*string)(unsafe.Pointer(&b))
cursor++
s.cursor = cursor
return nil, key, nil
case '\\':
cursor++
if char(p, cursor) == nul {
s.cursor = cursor
if !s.read() {
return nil, "", errors.ErrUnexpectedEndOfJSON("string", s.totalOffset())
}
buf, cursor, p = s.statForRetry()
}
case nul:
s.cursor = cursor
if !s.read() {
return nil, "", errors.ErrUnexpectedEndOfJSON("string", s.totalOffset())
}
buf, cursor, p = s.statForRetry()
}
}
}
func decodeKeyStream(d *structDecoder, s *Stream) (*structFieldSet, string, error) {
key, err := d.stringDecoder.decodeStreamByte(s)
if err != nil {
return nil, "", err
}
k := *(*string)(unsafe.Pointer(&key))
return d.fieldMap[k], k, nil
}
func (d *structDecoder) DecodeStream(s *Stream, depth int64, p unsafe.Pointer) error {
depth++
if depth > maxDecodeNestingDepth {
return errors.ErrExceededMaxDepth(s.char(), s.cursor)
}
c := s.skipWhiteSpace()
switch c {
case 'n':
if err := nullBytes(s); err != nil {
return err
}
return nil
default:
if s.char() != '{' {
return errors.ErrNotAtBeginningOfValue(s.totalOffset())
}
}
s.cursor++
if s.skipWhiteSpace() == '}' {
s.cursor++
return nil
}
var (
seenFields map[int]struct{}
seenFieldNum int
)
firstWin := (s.Option.Flags & FirstWinOption) != 0
if firstWin {
seenFields = make(map[int]struct{}, d.fieldUniqueNameNum)
}
for {
s.reset()
field, key, err := d.keyStreamDecoder(d, s)
if err != nil {
return err
}
if s.skipWhiteSpace() != ':' {
return errors.ErrExpected("colon after object key", s.totalOffset())
}
s.cursor++
if field != nil {
if field.err != nil {
return field.err
}
if firstWin {
if _, exists := seenFields[field.fieldIdx]; exists {
if err := s.skipValue(depth); err != nil {
return err
}
} else {
if err := field.dec.DecodeStream(s, depth, unsafe.Pointer(uintptr(p)+field.offset)); err != nil {
return err
}
seenFieldNum++
if d.fieldUniqueNameNum <= seenFieldNum {
return s.skipObject(depth)
}
seenFields[field.fieldIdx] = struct{}{}
}
} else {
if err := field.dec.DecodeStream(s, depth, unsafe.Pointer(uintptr(p)+field.offset)); err != nil {
return err
}
}
} else if s.DisallowUnknownFields {
return fmt.Errorf("json: unknown field %q", key)
} else {
if err := s.skipValue(depth); err != nil {
return err
}
}
c := s.skipWhiteSpace()
if c == '}' {
s.cursor++
return nil
}
if c != ',' {
return errors.ErrExpected("comma after object element", s.totalOffset())
}
s.cursor++
}
}
func (d *structDecoder) Decode(ctx *RuntimeContext, cursor, depth int64, p unsafe.Pointer) (int64, error) {
buf := ctx.Buf
depth++
if depth > maxDecodeNestingDepth {
return 0, errors.ErrExceededMaxDepth(buf[cursor], cursor)
}
buflen := int64(len(buf))
cursor = skipWhiteSpace(buf, cursor)
b := (*sliceHeader)(unsafe.Pointer(&buf)).data
switch char(b, cursor) {
case 'n':
if err := validateNull(buf, cursor); err != nil {
return 0, err
}
cursor += 4
return cursor, nil
case '{':
default:
return 0, errors.ErrNotAtBeginningOfValue(cursor)
}
cursor++
cursor = skipWhiteSpace(buf, cursor)
if buf[cursor] == '}' {
cursor++
return cursor, nil
}
var (
seenFields map[int]struct{}
seenFieldNum int
)
firstWin := (ctx.Option.Flags & FirstWinOption) != 0
if firstWin {
seenFields = make(map[int]struct{}, d.fieldUniqueNameNum)
}
for {
c, field, err := d.keyDecoder(d, buf, cursor)
if err != nil {
return 0, err
}
cursor = skipWhiteSpace(buf, c)
if char(b, cursor) != ':' {
return 0, errors.ErrExpected("colon after object key", cursor)
}
cursor++
if cursor >= buflen {
return 0, errors.ErrExpected("object value after colon", cursor)
}
if field != nil {
if field.err != nil {
return 0, field.err
}
if firstWin {
if _, exists := seenFields[field.fieldIdx]; exists {
c, err := skipValue(buf, cursor, depth)
if err != nil {
return 0, err
}
cursor = c
} else {
c, err := field.dec.Decode(ctx, cursor, depth, unsafe.Pointer(uintptr(p)+field.offset))
if err != nil {
return 0, err
}
cursor = c
seenFieldNum++
if d.fieldUniqueNameNum <= seenFieldNum {
return skipObject(buf, cursor, depth)
}
seenFields[field.fieldIdx] = struct{}{}
}
} else {
c, err := field.dec.Decode(ctx, cursor, depth, unsafe.Pointer(uintptr(p)+field.offset))
if err != nil {
return 0, err
}
cursor = c
}
} else {
c, err := skipValue(buf, cursor, depth)
if err != nil {
return 0, err
}
cursor = c
}
cursor = skipWhiteSpace(buf, cursor)
if char(b, cursor) == '}' {
cursor++
return cursor, nil
}
if char(b, cursor) != ',' {
return 0, errors.ErrExpected("comma after object element", cursor)
}
cursor++
}
}

View File

@@ -0,0 +1,29 @@
package decoder
import (
"context"
"encoding"
"encoding/json"
"reflect"
"unsafe"
)
type Decoder interface {
Decode(*RuntimeContext, int64, int64, unsafe.Pointer) (int64, error)
DecodeStream(*Stream, int64, unsafe.Pointer) error
}
const (
nul = '\000'
maxDecodeNestingDepth = 10000
)
type unmarshalerContext interface {
UnmarshalJSON(context.Context, []byte) error
}
var (
unmarshalJSONType = reflect.TypeOf((*json.Unmarshaler)(nil)).Elem()
unmarshalJSONContextType = reflect.TypeOf((*unmarshalerContext)(nil)).Elem()
unmarshalTextType = reflect.TypeOf((*encoding.TextUnmarshaler)(nil)).Elem()
)

View File

@@ -0,0 +1,190 @@
package decoder
import (
"fmt"
"reflect"
"unsafe"
"github.com/goccy/go-json/internal/errors"
"github.com/goccy/go-json/internal/runtime"
)
type uintDecoder struct {
typ *runtime.Type
kind reflect.Kind
op func(unsafe.Pointer, uint64)
structName string
fieldName string
}
func newUintDecoder(typ *runtime.Type, structName, fieldName string, op func(unsafe.Pointer, uint64)) *uintDecoder {
return &uintDecoder{
typ: typ,
kind: typ.Kind(),
op: op,
structName: structName,
fieldName: fieldName,
}
}
func (d *uintDecoder) typeError(buf []byte, offset int64) *errors.UnmarshalTypeError {
return &errors.UnmarshalTypeError{
Value: fmt.Sprintf("number %s", string(buf)),
Type: runtime.RType2Type(d.typ),
Offset: offset,
}
}
var (
pow10u64 = [...]uint64{
1e00, 1e01, 1e02, 1e03, 1e04, 1e05, 1e06, 1e07, 1e08, 1e09,
1e10, 1e11, 1e12, 1e13, 1e14, 1e15, 1e16, 1e17, 1e18, 1e19,
}
pow10u64Len = len(pow10u64)
)
func (d *uintDecoder) parseUint(b []byte) (uint64, error) {
maxDigit := len(b)
if maxDigit > pow10u64Len {
return 0, fmt.Errorf("invalid length of number")
}
sum := uint64(0)
for i := 0; i < maxDigit; i++ {
c := uint64(b[i]) - 48
digitValue := pow10u64[maxDigit-i-1]
sum += c * digitValue
}
return sum, nil
}
func (d *uintDecoder) decodeStreamByte(s *Stream) ([]byte, error) {
for {
switch s.char() {
case ' ', '\n', '\t', '\r':
s.cursor++
continue
case '0':
s.cursor++
return numZeroBuf, nil
case '1', '2', '3', '4', '5', '6', '7', '8', '9':
start := s.cursor
for {
s.cursor++
if numTable[s.char()] {
continue
} else if s.char() == nul {
if s.read() {
s.cursor-- // for retry current character
continue
}
}
break
}
num := s.buf[start:s.cursor]
return num, nil
case 'n':
if err := nullBytes(s); err != nil {
return nil, err
}
return nil, nil
case nul:
if s.read() {
continue
}
default:
return nil, d.typeError([]byte{s.char()}, s.totalOffset())
}
break
}
return nil, errors.ErrUnexpectedEndOfJSON("number(unsigned integer)", s.totalOffset())
}
func (d *uintDecoder) decodeByte(buf []byte, cursor int64) ([]byte, int64, error) {
for {
switch buf[cursor] {
case ' ', '\n', '\t', '\r':
cursor++
continue
case '0':
cursor++
return numZeroBuf, cursor, nil
case '1', '2', '3', '4', '5', '6', '7', '8', '9':
start := cursor
cursor++
for numTable[buf[cursor]] {
cursor++
}
num := buf[start:cursor]
return num, cursor, nil
case 'n':
if err := validateNull(buf, cursor); err != nil {
return nil, 0, err
}
cursor += 4
return nil, cursor, nil
default:
return nil, 0, d.typeError([]byte{buf[cursor]}, cursor)
}
}
}
func (d *uintDecoder) DecodeStream(s *Stream, depth int64, p unsafe.Pointer) error {
bytes, err := d.decodeStreamByte(s)
if err != nil {
return err
}
if bytes == nil {
return nil
}
u64, err := d.parseUint(bytes)
if err != nil {
return d.typeError(bytes, s.totalOffset())
}
switch d.kind {
case reflect.Uint8:
if (1 << 8) <= u64 {
return d.typeError(bytes, s.totalOffset())
}
case reflect.Uint16:
if (1 << 16) <= u64 {
return d.typeError(bytes, s.totalOffset())
}
case reflect.Uint32:
if (1 << 32) <= u64 {
return d.typeError(bytes, s.totalOffset())
}
}
d.op(p, u64)
return nil
}
func (d *uintDecoder) Decode(ctx *RuntimeContext, cursor, depth int64, p unsafe.Pointer) (int64, error) {
bytes, c, err := d.decodeByte(ctx.Buf, cursor)
if err != nil {
return 0, err
}
if bytes == nil {
return c, nil
}
cursor = c
u64, err := d.parseUint(bytes)
if err != nil {
return 0, d.typeError(bytes, cursor)
}
switch d.kind {
case reflect.Uint8:
if (1 << 8) <= u64 {
return 0, d.typeError(bytes, cursor)
}
case reflect.Uint16:
if (1 << 16) <= u64 {
return 0, d.typeError(bytes, cursor)
}
case reflect.Uint32:
if (1 << 32) <= u64 {
return 0, d.typeError(bytes, cursor)
}
}
d.op(p, u64)
return cursor, nil
}

View File

@@ -0,0 +1,91 @@
package decoder
import (
"encoding/json"
"unsafe"
"github.com/goccy/go-json/internal/errors"
"github.com/goccy/go-json/internal/runtime"
)
type unmarshalJSONDecoder struct {
typ *runtime.Type
structName string
fieldName string
}
func newUnmarshalJSONDecoder(typ *runtime.Type, structName, fieldName string) *unmarshalJSONDecoder {
return &unmarshalJSONDecoder{
typ: typ,
structName: structName,
fieldName: fieldName,
}
}
func (d *unmarshalJSONDecoder) annotateError(cursor int64, err error) {
switch e := err.(type) {
case *errors.UnmarshalTypeError:
e.Struct = d.structName
e.Field = d.fieldName
case *errors.SyntaxError:
e.Offset = cursor
}
}
func (d *unmarshalJSONDecoder) DecodeStream(s *Stream, depth int64, p unsafe.Pointer) error {
s.skipWhiteSpace()
start := s.cursor
if err := s.skipValue(depth); err != nil {
return err
}
src := s.buf[start:s.cursor]
dst := make([]byte, len(src))
copy(dst, src)
v := *(*interface{})(unsafe.Pointer(&emptyInterface{
typ: d.typ,
ptr: p,
}))
if (s.Option.Flags & ContextOption) != 0 {
if err := v.(unmarshalerContext).UnmarshalJSON(s.Option.Context, dst); err != nil {
d.annotateError(s.cursor, err)
return err
}
} else {
if err := v.(json.Unmarshaler).UnmarshalJSON(dst); err != nil {
d.annotateError(s.cursor, err)
return err
}
}
return nil
}
func (d *unmarshalJSONDecoder) Decode(ctx *RuntimeContext, cursor, depth int64, p unsafe.Pointer) (int64, error) {
buf := ctx.Buf
cursor = skipWhiteSpace(buf, cursor)
start := cursor
end, err := skipValue(buf, cursor, depth)
if err != nil {
return 0, err
}
src := buf[start:end]
dst := make([]byte, len(src))
copy(dst, src)
v := *(*interface{})(unsafe.Pointer(&emptyInterface{
typ: d.typ,
ptr: p,
}))
if (ctx.Option.Flags & ContextOption) != 0 {
if err := v.(unmarshalerContext).UnmarshalJSON(ctx.Option.Context, dst); err != nil {
d.annotateError(cursor, err)
return 0, err
}
} else {
if err := v.(json.Unmarshaler).UnmarshalJSON(dst); err != nil {
d.annotateError(cursor, err)
return 0, err
}
}
return end, nil
}

View File

@@ -0,0 +1,280 @@
package decoder
import (
"bytes"
"encoding"
"unicode"
"unicode/utf16"
"unicode/utf8"
"unsafe"
"github.com/goccy/go-json/internal/errors"
"github.com/goccy/go-json/internal/runtime"
)
type unmarshalTextDecoder struct {
typ *runtime.Type
structName string
fieldName string
}
func newUnmarshalTextDecoder(typ *runtime.Type, structName, fieldName string) *unmarshalTextDecoder {
return &unmarshalTextDecoder{
typ: typ,
structName: structName,
fieldName: fieldName,
}
}
func (d *unmarshalTextDecoder) annotateError(cursor int64, err error) {
switch e := err.(type) {
case *errors.UnmarshalTypeError:
e.Struct = d.structName
e.Field = d.fieldName
case *errors.SyntaxError:
e.Offset = cursor
}
}
var (
nullbytes = []byte(`null`)
)
func (d *unmarshalTextDecoder) DecodeStream(s *Stream, depth int64, p unsafe.Pointer) error {
s.skipWhiteSpace()
start := s.cursor
if err := s.skipValue(depth); err != nil {
return err
}
src := s.buf[start:s.cursor]
if len(src) > 0 {
switch src[0] {
case '[':
return &errors.UnmarshalTypeError{
Value: "array",
Type: runtime.RType2Type(d.typ),
Offset: s.totalOffset(),
}
case '{':
return &errors.UnmarshalTypeError{
Value: "object",
Type: runtime.RType2Type(d.typ),
Offset: s.totalOffset(),
}
case '-', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9':
return &errors.UnmarshalTypeError{
Value: "number",
Type: runtime.RType2Type(d.typ),
Offset: s.totalOffset(),
}
case 'n':
if bytes.Equal(src, nullbytes) {
*(*unsafe.Pointer)(p) = nil
return nil
}
}
}
dst := make([]byte, len(src))
copy(dst, src)
if b, ok := unquoteBytes(dst); ok {
dst = b
}
v := *(*interface{})(unsafe.Pointer(&emptyInterface{
typ: d.typ,
ptr: p,
}))
if err := v.(encoding.TextUnmarshaler).UnmarshalText(dst); err != nil {
d.annotateError(s.cursor, err)
return err
}
return nil
}
func (d *unmarshalTextDecoder) Decode(ctx *RuntimeContext, cursor, depth int64, p unsafe.Pointer) (int64, error) {
buf := ctx.Buf
cursor = skipWhiteSpace(buf, cursor)
start := cursor
end, err := skipValue(buf, cursor, depth)
if err != nil {
return 0, err
}
src := buf[start:end]
if len(src) > 0 {
switch src[0] {
case '[':
return 0, &errors.UnmarshalTypeError{
Value: "array",
Type: runtime.RType2Type(d.typ),
Offset: start,
}
case '{':
return 0, &errors.UnmarshalTypeError{
Value: "object",
Type: runtime.RType2Type(d.typ),
Offset: start,
}
case '-', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9':
return 0, &errors.UnmarshalTypeError{
Value: "number",
Type: runtime.RType2Type(d.typ),
Offset: start,
}
case 'n':
if bytes.Equal(src, nullbytes) {
*(*unsafe.Pointer)(p) = nil
return end, nil
}
}
}
if s, ok := unquoteBytes(src); ok {
src = s
}
v := *(*interface{})(unsafe.Pointer(&emptyInterface{
typ: d.typ,
ptr: *(*unsafe.Pointer)(unsafe.Pointer(&p)),
}))
if err := v.(encoding.TextUnmarshaler).UnmarshalText(src); err != nil {
d.annotateError(cursor, err)
return 0, err
}
return end, nil
}
func unquoteBytes(s []byte) (t []byte, ok bool) {
length := len(s)
if length < 2 || s[0] != '"' || s[length-1] != '"' {
return
}
s = s[1 : length-1]
length -= 2
// Check for unusual characters. If there are none,
// then no unquoting is needed, so return a slice of the
// original bytes.
r := 0
for r < length {
c := s[r]
if c == '\\' || c == '"' || c < ' ' {
break
}
if c < utf8.RuneSelf {
r++
continue
}
rr, size := utf8.DecodeRune(s[r:])
if rr == utf8.RuneError && size == 1 {
break
}
r += size
}
if r == length {
return s, true
}
b := make([]byte, length+2*utf8.UTFMax)
w := copy(b, s[0:r])
for r < length {
// Out of room? Can only happen if s is full of
// malformed UTF-8 and we're replacing each
// byte with RuneError.
if w >= len(b)-2*utf8.UTFMax {
nb := make([]byte, (len(b)+utf8.UTFMax)*2)
copy(nb, b[0:w])
b = nb
}
switch c := s[r]; {
case c == '\\':
r++
if r >= length {
return
}
switch s[r] {
default:
return
case '"', '\\', '/', '\'':
b[w] = s[r]
r++
w++
case 'b':
b[w] = '\b'
r++
w++
case 'f':
b[w] = '\f'
r++
w++
case 'n':
b[w] = '\n'
r++
w++
case 'r':
b[w] = '\r'
r++
w++
case 't':
b[w] = '\t'
r++
w++
case 'u':
r--
rr := getu4(s[r:])
if rr < 0 {
return
}
r += 6
if utf16.IsSurrogate(rr) {
rr1 := getu4(s[r:])
if dec := utf16.DecodeRune(rr, rr1); dec != unicode.ReplacementChar {
// A valid pair; consume.
r += 6
w += utf8.EncodeRune(b[w:], dec)
break
}
// Invalid surrogate; fall back to replacement rune.
rr = unicode.ReplacementChar
}
w += utf8.EncodeRune(b[w:], rr)
}
// Quote, control characters are invalid.
case c == '"', c < ' ':
return
// ASCII
case c < utf8.RuneSelf:
b[w] = c
r++
w++
// Coerce to well-formed UTF-8.
default:
rr, size := utf8.DecodeRune(s[r:])
r += size
w += utf8.EncodeRune(b[w:], rr)
}
}
return b[0:w], true
}
func getu4(s []byte) rune {
if len(s) < 6 || s[0] != '\\' || s[1] != 'u' {
return -1
}
var r rune
for _, c := range s[2:6] {
switch {
case '0' <= c && c <= '9':
c = c - '0'
case 'a' <= c && c <= 'f':
c = c - 'a' + 10
case 'A' <= c && c <= 'F':
c = c - 'A' + 10
default:
return -1
}
r = r*16 + rune(c)
}
return r
}

View File

@@ -0,0 +1,68 @@
package decoder
import (
"reflect"
"unsafe"
"github.com/goccy/go-json/internal/runtime"
)
type wrappedStringDecoder struct {
typ *runtime.Type
dec Decoder
stringDecoder *stringDecoder
structName string
fieldName string
isPtrType bool
}
func newWrappedStringDecoder(typ *runtime.Type, dec Decoder, structName, fieldName string) *wrappedStringDecoder {
return &wrappedStringDecoder{
typ: typ,
dec: dec,
stringDecoder: newStringDecoder(structName, fieldName),
structName: structName,
fieldName: fieldName,
isPtrType: typ.Kind() == reflect.Ptr,
}
}
func (d *wrappedStringDecoder) DecodeStream(s *Stream, depth int64, p unsafe.Pointer) error {
bytes, err := d.stringDecoder.decodeStreamByte(s)
if err != nil {
return err
}
if bytes == nil {
if d.isPtrType {
*(*unsafe.Pointer)(p) = nil
}
return nil
}
b := make([]byte, len(bytes)+1)
copy(b, bytes)
if _, err := d.dec.Decode(&RuntimeContext{Buf: b}, 0, depth, p); err != nil {
return err
}
return nil
}
func (d *wrappedStringDecoder) Decode(ctx *RuntimeContext, cursor, depth int64, p unsafe.Pointer) (int64, error) {
bytes, c, err := d.stringDecoder.decodeByte(ctx.Buf, cursor)
if err != nil {
return 0, err
}
if bytes == nil {
if d.isPtrType {
*(*unsafe.Pointer)(p) = nil
}
return c, nil
}
bytes = append(bytes, nul)
oldBuf := ctx.Buf
ctx.Buf = bytes
if _, err := d.dec.Decode(ctx, 0, depth, p); err != nil {
return 0, err
}
ctx.Buf = oldBuf
return c, nil
}

View File

@@ -0,0 +1,286 @@
package encoder
import (
"bytes"
"fmt"
"strconv"
"unsafe"
"github.com/goccy/go-json/internal/errors"
)
var (
isWhiteSpace = [256]bool{
' ': true,
'\n': true,
'\t': true,
'\r': true,
}
isHTMLEscapeChar = [256]bool{
'<': true,
'>': true,
'&': true,
}
nul = byte('\000')
)
func Compact(buf *bytes.Buffer, src []byte, escape bool) error {
if len(src) == 0 {
return errors.ErrUnexpectedEndOfJSON("", 0)
}
buf.Grow(len(src))
dst := buf.Bytes()
ctx := TakeRuntimeContext()
ctxBuf := ctx.Buf[:0]
ctxBuf = append(append(ctxBuf, src...), nul)
ctx.Buf = ctxBuf
if err := compactAndWrite(buf, dst, ctxBuf, escape); err != nil {
ReleaseRuntimeContext(ctx)
return err
}
ReleaseRuntimeContext(ctx)
return nil
}
func compactAndWrite(buf *bytes.Buffer, dst []byte, src []byte, escape bool) error {
dst, err := compact(dst, src, escape)
if err != nil {
return err
}
if _, err := buf.Write(dst); err != nil {
return err
}
return nil
}
func compact(dst, src []byte, escape bool) ([]byte, error) {
buf, cursor, err := compactValue(dst, src, 0, escape)
if err != nil {
return nil, err
}
if err := validateEndBuf(src, cursor); err != nil {
return nil, err
}
return buf, nil
}
func validateEndBuf(src []byte, cursor int64) error {
for {
switch src[cursor] {
case ' ', '\t', '\n', '\r':
cursor++
continue
case nul:
return nil
}
return errors.ErrSyntax(
fmt.Sprintf("invalid character '%c' after top-level value", src[cursor]),
cursor+1,
)
}
}
func skipWhiteSpace(buf []byte, cursor int64) int64 {
LOOP:
if isWhiteSpace[buf[cursor]] {
cursor++
goto LOOP
}
return cursor
}
func compactValue(dst, src []byte, cursor int64, escape bool) ([]byte, int64, error) {
for {
switch src[cursor] {
case ' ', '\t', '\n', '\r':
cursor++
continue
case '{':
return compactObject(dst, src, cursor, escape)
case '}':
return nil, 0, errors.ErrSyntax("unexpected character '}'", cursor)
case '[':
return compactArray(dst, src, cursor, escape)
case ']':
return nil, 0, errors.ErrSyntax("unexpected character ']'", cursor)
case '"':
return compactString(dst, src, cursor, escape)
case '-', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9':
return compactNumber(dst, src, cursor)
case 't':
return compactTrue(dst, src, cursor)
case 'f':
return compactFalse(dst, src, cursor)
case 'n':
return compactNull(dst, src, cursor)
default:
return nil, 0, errors.ErrSyntax(fmt.Sprintf("unexpected character '%c'", src[cursor]), cursor)
}
}
}
func compactObject(dst, src []byte, cursor int64, escape bool) ([]byte, int64, error) {
if src[cursor] == '{' {
dst = append(dst, '{')
} else {
return nil, 0, errors.ErrExpected("expected { character for object value", cursor)
}
cursor = skipWhiteSpace(src, cursor+1)
if src[cursor] == '}' {
dst = append(dst, '}')
return dst, cursor + 1, nil
}
var err error
for {
cursor = skipWhiteSpace(src, cursor)
dst, cursor, err = compactString(dst, src, cursor, escape)
if err != nil {
return nil, 0, err
}
cursor = skipWhiteSpace(src, cursor)
if src[cursor] != ':' {
return nil, 0, errors.ErrExpected("colon after object key", cursor)
}
dst = append(dst, ':')
dst, cursor, err = compactValue(dst, src, cursor+1, escape)
if err != nil {
return nil, 0, err
}
cursor = skipWhiteSpace(src, cursor)
switch src[cursor] {
case '}':
dst = append(dst, '}')
cursor++
return dst, cursor, nil
case ',':
dst = append(dst, ',')
default:
return nil, 0, errors.ErrExpected("comma after object value", cursor)
}
cursor++
}
}
func compactArray(dst, src []byte, cursor int64, escape bool) ([]byte, int64, error) {
if src[cursor] == '[' {
dst = append(dst, '[')
} else {
return nil, 0, errors.ErrExpected("expected [ character for array value", cursor)
}
cursor = skipWhiteSpace(src, cursor+1)
if src[cursor] == ']' {
dst = append(dst, ']')
return dst, cursor + 1, nil
}
var err error
for {
dst, cursor, err = compactValue(dst, src, cursor, escape)
if err != nil {
return nil, 0, err
}
cursor = skipWhiteSpace(src, cursor)
switch src[cursor] {
case ']':
dst = append(dst, ']')
cursor++
return dst, cursor, nil
case ',':
dst = append(dst, ',')
default:
return nil, 0, errors.ErrExpected("comma after array value", cursor)
}
cursor++
}
}
func compactString(dst, src []byte, cursor int64, escape bool) ([]byte, int64, error) {
if src[cursor] != '"' {
return nil, 0, errors.ErrInvalidCharacter(src[cursor], "string", cursor)
}
start := cursor
for {
cursor++
c := src[cursor]
if escape {
if isHTMLEscapeChar[c] {
dst = append(dst, src[start:cursor]...)
dst = append(dst, `\u00`...)
dst = append(dst, hex[c>>4], hex[c&0xF])
start = cursor + 1
} else if c == 0xE2 && cursor+2 < int64(len(src)) && src[cursor+1] == 0x80 && src[cursor+2]&^1 == 0xA8 {
dst = append(dst, src[start:cursor]...)
dst = append(dst, `\u202`...)
dst = append(dst, hex[src[cursor+2]&0xF])
cursor += 2
start = cursor + 3
}
}
switch c {
case '\\':
cursor++
if src[cursor] == nul {
return nil, 0, errors.ErrUnexpectedEndOfJSON("string", int64(len(src)))
}
case '"':
cursor++
return append(dst, src[start:cursor]...), cursor, nil
case nul:
return nil, 0, errors.ErrUnexpectedEndOfJSON("string", int64(len(src)))
}
}
}
func compactNumber(dst, src []byte, cursor int64) ([]byte, int64, error) {
start := cursor
for {
cursor++
if floatTable[src[cursor]] {
continue
}
break
}
num := src[start:cursor]
if _, err := strconv.ParseFloat(*(*string)(unsafe.Pointer(&num)), 64); err != nil {
return nil, 0, err
}
dst = append(dst, num...)
return dst, cursor, nil
}
func compactTrue(dst, src []byte, cursor int64) ([]byte, int64, error) {
if cursor+3 >= int64(len(src)) {
return nil, 0, errors.ErrUnexpectedEndOfJSON("true", cursor)
}
if !bytes.Equal(src[cursor:cursor+4], []byte(`true`)) {
return nil, 0, errors.ErrInvalidCharacter(src[cursor], "true", cursor)
}
dst = append(dst, "true"...)
cursor += 4
return dst, cursor, nil
}
func compactFalse(dst, src []byte, cursor int64) ([]byte, int64, error) {
if cursor+4 >= int64(len(src)) {
return nil, 0, errors.ErrUnexpectedEndOfJSON("false", cursor)
}
if !bytes.Equal(src[cursor:cursor+5], []byte(`false`)) {
return nil, 0, errors.ErrInvalidCharacter(src[cursor], "false", cursor)
}
dst = append(dst, "false"...)
cursor += 5
return dst, cursor, nil
}
func compactNull(dst, src []byte, cursor int64) ([]byte, int64, error) {
if cursor+3 >= int64(len(src)) {
return nil, 0, errors.ErrUnexpectedEndOfJSON("null", cursor)
}
if !bytes.Equal(src[cursor:cursor+4], []byte(`null`)) {
return nil, 0, errors.ErrInvalidCharacter(src[cursor], "null", cursor)
}
dst = append(dst, "null"...)
cursor += 4
return dst, cursor, nil
}

Some files were not shown because too many files have changed in this diff Show More