Changelog 1.15.5 (#17392)

* SECURITY
  * Upgrade Bluemonday to v1.0.16 (#17372) (#17374)
  * Ensure correct SSH permissions check for private and restricted users (#17370) (#17373)
* BUGFIXES
  * Prevent NPE in CSV diff rendering when column removed (#17018) (#17377)
  * Offer rsa-sha2-512 and rsa-sha2-256 algorithms in internal SSH (#17281) (#17376)
  * Don't panic if we fail to parse U2FRegistration data (#17304) (#17371)
  * Ensure popup text is aligned left (backport for 1.15) (#17343)
  * Ensure that git daemon export ok is created for mirrors (#17243) (#17306)
  * Disable core.protectNTFS (#17300) (#17302)
  * Use pointer for wrappedConn methods (#17295) (#17296)
  * AutoRegistration is supposed to be working with disabled registration (backport) (#17292)
  * Handle duplicate keys on GPG key ring (#17242) (#17284)
  * Fix SVG side by side comparison link (#17375) (#17391)

Signed-off-by: Andrew Thornton <art27@cantab.net>

.drone.yml

@@ -527,7 +527,7 @@ steps:
 
   - name: release-branch
     pull: always
-    image: plugins/s3:1
+    image: woodpeckerci/plugin-s3:latest
     settings:
       acl: public-read
       bucket: gitea-artifacts
@@ -548,7 +548,7 @@ steps:
         - push
 
   - name: release-main
-    image: plugins/s3:1
+    image: woodpeckerci/plugin-s3:latest
     settings:
       acl: public-read
       bucket: gitea-artifacts
@@ -623,7 +623,7 @@ steps:
 
   - name: release-tag
     pull: always
-    image: plugins/s3:1
+    image: woodpeckerci/plugin-s3:latest
     settings:
       acl: public-read
       bucket: gitea-artifacts

CHANGELOG.md

@@ -4,6 +4,74 @@ This changelog goes through all the changes that have been made in each release
 without substantial changes to our git log; to see the highlights of what has
 been added to each release, please refer to the [blog](https://blog.gitea.io).
 
+## [1.15.5](https://github.com/go-gitea/gitea/releases/tag/v1.15.5) - 2021-10-21
+
+* SECURITY
+  * Upgrade Bluemonday to v1.0.16 (#17372) (#17374)
+  * Ensure correct SSH permissions check for private and restricted users (#17370) (#17373)
+* BUGFIXES
+  * Prevent NPE in CSV diff rendering when column removed (#17018) (#17377)
+  * Offer rsa-sha2-512 and rsa-sha2-256 algorithms in internal SSH (#17281) (#17376)
+  * Don't panic if we fail to parse U2FRegistration data (#17304) (#17371)
+  * Ensure popup text is aligned left (backport for 1.15) (#17343)
+  * Ensure that git daemon export ok is created for mirrors (#17243) (#17306)
+  * Disable core.protectNTFS (#17300) (#17302)
+  * Use pointer for wrappedConn methods (#17295) (#17296)
+  * AutoRegistration is supposed to be working with disabled registration (backport) (#17292)
+  * Handle duplicate keys on GPG key ring (#17242) (#17284)
+  * Fix SVG side by side comparison link (#17375) (#17391)
+
+## [1.15.4](https://github.com/go-gitea/gitea/releases/tag/v1.15.4) - 2021-10-08
+* BUGFIXES
+  * Raw file API: don't try to interpret 40char filenames as commit SHA (#17185) (#17272)
+  * Don't allow merged PRs to be reopened (#17192) (#17271)
+  * Fix incorrect repository count on organization tab of dashboard (#17256) (#17266)
+  * Fix unwanted team review request deletion (#17257) (#17264)
+  * Fix broken Activities link in team dashboard (#17255) (#17258)
+  * API pull's head/base have correct permission(#17214) (#17245)
+  * Fix stange behavior of DownloadPullDiffOrPatch in incorect index (#17223) (#17227)
+  * Upgrade xorm to v1.2.5 (#17177) (#17188)
+  * Fix missing repo link in issue/pull assigned emails (#17183) (#17184)
+  * Fix bug of get context user (#17169) (#17172)
+  * Nicely handle missing user in collaborations (#17049) (#17166)
+  * Add Horizontal scrollbar to inner menu on Chrome (#17086) (#17164)
+  * Fix wrong i18n keys (#17150) (#17153)
+  * Fix Archive Creation: correct transaction ending (#17151)
+  * Prevent panic in Org mode HighlightCodeBlock (#17140) (#17141)
+  * Create doctor command to fix repo_units broken by dumps from 1.14.3-1.14.6 (#17136) (#17137)
+* ENHANCEMENT
+  * Check user instead of organization when creating a repo from a template via API (#16346) (#17195)
+* TRANSLATION
+  * v1.15 fix Sprintf format 'verbs' in locale files (#17187)
+
+
+## [1.15.3](https://github.com/go-gitea/gitea/releases/tag/v1.15.3) - 2021-09-19
+
+* ENHANCEMENTS
+  * Add fluid to ui container class to remove margin (#16396) (#16976)
+  * Add caller to cat-file batch calls (#17082) (#17089)
+* BUGFIXES
+  * Render full plain readme. (#17083) (#17090)
+  * Upgrade xorm to v1.2.4 (#17059)
+  * Fix bug of migrate comments which only fetch one page (#17055) (#17058)
+  * Do not show issue context popup on external issues (#17050) (#17054)
+  * Decrement Fork Num when converting from Fork (#17035) (#17046)
+  * Correctly rollback in ForkRepository (#17034) (#17045)
+  * Fix missing close in WalkGitLog (#17008) (#17009)
+  * Add prefix to SVG id/class attributes (#16997) (#17000)
+  * Fix bug of migrated repository not index (#16991) (#16996)
+  * Skip AllowedUserVisibilityModes validation on update user if it is an organisation (#16988) (#16990)
+  * Fix storage Iterate bug and Add storage doctor to delete garbage attachments (#16971) (#16977)
+  * Fix issue with issue default mail template (#16956) (#16975)
+  * Ensure that rebase conflicts are handled in updates (#16952) (#16960)
+  * Prevent panic on diff generation (#16950) (#16951)
+
+## [1.15.2](https://github.com/go-gitea/gitea/releases/tag/v1.15.2) - 2021-09-03
+
+* BUGFIXES
+  * Add unique constraint back into issue_index (#16938)
+  * Close storage objects before cleaning (#16934) (#16942)
+
 ## [1.15.1](https://github.com/go-gitea/gitea/releases/tag/v1.15.1) - 2021-09-02
 
 * BUGFIXES

README.md

@@ -12,9 +12,6 @@
   <a href="https://discord.gg/Gitea" title="Join the Discord chat at https://discord.gg/Gitea">
     <img src="https://img.shields.io/discord/322538954119184384.svg">
   </a>
-  <a href="https://microbadger.com/images/gitea/gitea" title="Get your own image badge on microbadger.com">
-    <img src="https://images.microbadger.com/badges/image/gitea/gitea.svg">
-  </a>
   <a href="https://codecov.io/gh/go-gitea/gitea" title="Codecov">
     <img src="https://codecov.io/gh/go-gitea/gitea/branch/main/graph/badge.svg">
   </a>

build/generate-svg.js

@@ -29,6 +29,7 @@ async function processFile(file, {prefix, fullName} = {}) {
     plugins: extendDefaultPlugins([
       'removeXMLNS',
       'removeDimensions',
+      {name: 'prefixIds', params: {prefix: () => name}},
       {
         name: 'addClassesToSVGElement',
         params: {classNames: ['svg', name]},

cmd/doctor.go

@@ -124,7 +124,6 @@ func runRecreateTable(ctx *cli.Context) error {
 }
 
 func runDoctor(ctx *cli.Context) error {
-
 	// Silence the default loggers
 	log.DelNamedLogger("console")
 	log.DelNamedLogger(log.DEFAULT)

custom/conf/app.example.ini

@@ -576,6 +576,8 @@ PATH =
 ;;
 ;; (Go-Git only) Don't cache objects greater than this in memory. (Set to 0 to disable.)
 ;LARGE_OBJECT_THRESHOLD = 1048576
+;; Set to true to forcibly set core.protectNTFS=false
+;DISABLE_CORE_PROTECT_NTFS=false
 
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

docs/content/doc/advanced/config-cheat-sheet.en-us.md

@@ -839,6 +839,7 @@ NB: You must have `DISABLE_ROUTER_LOG` set to `false` for this option to take ef
 - `VERBOSE_PUSH`: **true**: Print status information about pushes as they are being processed.
 - `VERBOSE_PUSH_DELAY`: **5s**: Only print verbose information if push takes longer than this delay.
 - `LARGE_OBJECT_THRESHOLD`: **1048576**: (Go-Git only), don't cache objects greater than this in memory. (Set to 0 to disable.)
+- `DISABLE_CORE_PROTECT_NTFS`: **false** Set to true to forcibly set `core.protectNTFS` to false.
 ## Git - Timeout settings (`git.timeout`)
 - `DEFAUlT`: **360**: Git operations default timeout seconds.
 - `MIGRATE`: **600**: Migrate external repositories timeout seconds.

go.mod

@@ -80,7 +80,7 @@ require (
 	github.com/mattn/go-runewidth v0.0.13 // indirect
 	github.com/mattn/go-sqlite3 v1.14.8
 	github.com/mholt/archiver/v3 v3.5.0
-	github.com/microcosm-cc/bluemonday v1.0.15
+	github.com/microcosm-cc/bluemonday v1.0.16
 	github.com/miekg/dns v1.1.43 // indirect
 	github.com/minio/md5-simd v1.1.2 // indirect
 	github.com/minio/minio-go/v7 v7.0.12
@@ -125,7 +125,7 @@ require (
 	go.uber.org/multierr v1.7.0 // indirect
 	go.uber.org/zap v1.18.1 // indirect
 	golang.org/x/crypto v0.0.0-20210616213533-5ff15b29337e
-	golang.org/x/net v0.0.0-20210614182718-04defd469f4e
+	golang.org/x/net v0.0.0-20211020060615-d418f374d309
 	golang.org/x/oauth2 v0.0.0-20210628180205-a41e5a781914
 	golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c
 	golang.org/x/text v0.3.6
@@ -139,7 +139,7 @@ require (
 	mvdan.cc/xurls/v2 v2.2.0
 	strk.kbt.io/projects/go/libravatar v0.0.0-20191008002943-06d1c002b251
 	xorm.io/builder v0.3.9
-	xorm.io/xorm v1.2.2
+	xorm.io/xorm v1.2.5
 )
 
 replace github.com/hashicorp/go-version => github.com/6543/go-version v1.3.1

go.sum

@@ -868,8 +868,8 @@ github.com/mholt/acmez v0.1.3 h1:J7MmNIk4Qf9b8mAGqAh4XkNeowv3f1zW816yf4zt7Qk=
 github.com/mholt/acmez v0.1.3/go.mod h1:8qnn8QA/Ewx8E3ZSsmscqsIjhhpxuy9vqdgbX2ceceM=
 github.com/mholt/archiver/v3 v3.5.0 h1:nE8gZIrw66cu4osS/U7UW7YDuGMHssxKutU8IfWxwWE=
 github.com/mholt/archiver/v3 v3.5.0/go.mod h1:qqTTPUK/HZPFgFQ/TJ3BzvTpF/dPtFVJXdQbCmeMxwc=
-github.com/microcosm-cc/bluemonday v1.0.15 h1:J4uN+qPng9rvkBZBoBb8YGR+ijuklIMpSOZZLjYpbeY=
-github.com/microcosm-cc/bluemonday v1.0.15/go.mod h1:ZLvAzeakRwrGnzQEvstVzVt3ZpqOF2+sdFr0Om+ce30=
+github.com/microcosm-cc/bluemonday v1.0.16 h1:kHmAq2t7WPWLjiGvzKa5o3HzSfahUKiOq7fAPUiMNIc=
+github.com/microcosm-cc/bluemonday v1.0.16/go.mod h1:Z0r70sCuXHig8YpBzCc5eGHAap2K7e/u082ZUpDRRqM=
 github.com/miekg/dns v1.0.14/go.mod h1:W1PPwlIAgtquWBMBEV9nkV9Cazfe8ScdGz/Lj7v3Nrg=
 github.com/miekg/dns v1.1.42/go.mod h1:+evo5L0630/F6ca/Z9+GAqzhjGyn8/c+TBaOyfEl0V4=
 github.com/miekg/dns v1.1.43 h1:JKfpVSCB84vrAmHzyrsxB5NAr5kLoMXZArPSw7Qlgyg=
@@ -1364,8 +1364,9 @@ golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v
 golang.org/x/net v0.0.0-20210326060303-6b1517762897/go.mod h1:uSPa2vr4CLtc/ILN5odXGNXS6mhrKVzTaCXzk9m6W3k=
 golang.org/x/net v0.0.0-20210331060903-cb1fcc7394e5/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM=
 golang.org/x/net v0.0.0-20210525063256-abc453219eb5/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
-golang.org/x/net v0.0.0-20210614182718-04defd469f4e h1:XpT3nA5TvE525Ne3hInMh6+GETgn27Zfm9dxsThnX2Q=
 golang.org/x/net v0.0.0-20210614182718-04defd469f4e/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
+golang.org/x/net v0.0.0-20211020060615-d418f374d309 h1:A0lJIi+hcTR6aajJH4YqKWwohY4aW9RO7oRMcdv+HKI=
+golang.org/x/net v0.0.0-20211020060615-d418f374d309/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
 golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
 golang.org/x/oauth2 v0.0.0-20181106182150-f42d05182288/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
 golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
@@ -1765,5 +1766,5 @@ xorm.io/builder v0.3.7/go.mod h1:aUW0S9eb9VCaPohFCH3j7czOx1PMW3i1HrSzbLYGBSE=
 xorm.io/builder v0.3.9 h1:Sd65/LdWyO7LR8+Cbd+e7mm3sK/7U9k0jS3999IDHMc=
 xorm.io/builder v0.3.9/go.mod h1:aUW0S9eb9VCaPohFCH3j7czOx1PMW3i1HrSzbLYGBSE=
 xorm.io/xorm v1.0.6/go.mod h1:uF9EtbhODq5kNWxMbnBEj8hRRZnlcNSz2t2N7HW/+A4=
-xorm.io/xorm v1.2.2 h1:FFBOEvJ++8fYBA9cywf2sxDVmFktl1SpJzTAG1ab06Y=
-xorm.io/xorm v1.2.2/go.mod h1:fTG8tSjk6O1BYxwuohZUK+S1glnRycsCF05L1qQyEU0=
+xorm.io/xorm v1.2.5 h1:tqN7OhN8P9xi52qBb76I8m5maAJMz/SSbgK2RGPCPbo=
+xorm.io/xorm v1.2.5/go.mod h1:fTG8tSjk6O1BYxwuohZUK+S1glnRycsCF05L1qQyEU0=

integrations/api_user_heatmap_test.go

@@ -8,8 +8,10 @@ import (
 	"fmt"
 	"net/http"
 	"testing"
+	"time"
 
 	"code.gitea.io/gitea/models"
+	"code.gitea.io/gitea/modules/timeutil"
 
 	"github.com/stretchr/testify/assert"
 )
@@ -20,6 +22,10 @@ func TestUserHeatmap(t *testing.T) {
 	normalUsername := "user2"
 	session := loginUser(t, adminUsername)
 
+	var fakeNow = time.Date(2011, 10, 20, 0, 0, 0, 0, time.Local)
+	timeutil.Set(fakeNow)
+	defer timeutil.Unset()
+
 	urlStr := fmt.Sprintf("/api/v1/users/%s/heatmap", normalUsername)
 	req := NewRequest(t, "GET", urlStr)
 	resp := session.MakeRequest(t, req, http.StatusOK)

models/access.go

@@ -225,6 +225,9 @@ func (repo *Repository) refreshCollaboratorAccesses(e Engine, accessMap map[int6
 		return fmt.Errorf("getCollaborations: %v", err)
 	}
 	for _, c := range collaborators {
+		if c.User.IsGhost() {
+			continue
+		}
 		updateUserAccess(accessMap, c.User, c.Collaboration.Mode)
 	}
 	return nil

models/attachment.go

@@ -144,6 +144,11 @@ func GetAttachmentByUUID(uuid string) (*Attachment, error) {
 	return getAttachmentByUUID(x, uuid)
 }
 
+// ExistAttachmentsByUUID returns true if attachment is exist by given UUID
+func ExistAttachmentsByUUID(uuid string) (bool, error) {
+	return x.Where("`uuid`=?", uuid).Exist(new(Attachment))
+}
+
 // GetAttachmentByReleaseIDFileName returns attachment by given releaseId and fileName.
 func GetAttachmentByReleaseIDFileName(releaseID int64, fileName string) (*Attachment, error) {
 	return getAttachmentByReleaseIDFileName(x, releaseID, fileName)

models/context.go

@@ -45,19 +45,16 @@ func WithContext(f func(ctx DBContext) error) error {
 // WithTx represents executing database operations on a transaction
 func WithTx(f func(ctx DBContext) error) error {
 	sess := x.NewSession()
+	defer sess.Close()
 	if err := sess.Begin(); err != nil {
-		sess.Close()
 		return err
 	}
 
 	if err := f(DBContext{sess}); err != nil {
-		sess.Close()
 		return err
 	}
 
-	err := sess.Commit()
-	sess.Close()
-	return err
+	return sess.Commit()
 }
 
 // Iterate iterates the databases and doing something

models/gpg_key_add.go

@@ -99,6 +99,46 @@ func AddGPGKey(ownerID int64, content, token, signature string) ([]*GPGKey, erro
 		verified = true
 	}
 
+	if len(ekeys) > 1 {
+		id2key := map[string]*openpgp.Entity{}
+		newEKeys := make([]*openpgp.Entity, 0, len(ekeys))
+		for _, ekey := range ekeys {
+			id := ekey.PrimaryKey.KeyIdString()
+			if original, has := id2key[id]; has {
+				// Coalesce this with the other one
+				for _, subkey := range ekey.Subkeys {
+					if subkey.PublicKey == nil {
+						continue
+					}
+					found := false
+
+					for _, originalSubkey := range original.Subkeys {
+						if originalSubkey.PublicKey == nil {
+							continue
+						}
+						if originalSubkey.PublicKey.KeyId == subkey.PublicKey.KeyId {
+							found = true
+							break
+						}
+					}
+					if !found {
+						original.Subkeys = append(original.Subkeys, subkey)
+					}
+				}
+				for name, identity := range ekey.Identities {
+					if _, has := original.Identities[name]; has {
+						continue
+					}
+					original.Identities[name] = identity
+				}
+				continue
+			}
+			id2key[id] = ekey
+			newEKeys = append(newEKeys, ekey)
+		}
+		ekeys = newEKeys
+	}
+
 	for _, ekey := range ekeys {
 		// Key ID cannot be duplicated.
 		has, err := sess.Where("key_id=?", ekey.PrimaryKey.KeyIdString()).

models/index.go

@@ -14,7 +14,7 @@ import (
 // ResourceIndex represents a resource index which could be used as issue/release and others
 // We can create different tables i.e. issue_index, release_index and etc.
 type ResourceIndex struct {
-	GroupID  int64 `xorm:"pk"`
+	GroupID  int64 `xorm:"pk unique"`
 	MaxIndex int64 `xorm:"index"`
 }
 

models/login_source.go

@@ -71,9 +71,9 @@ var (
 	_ convert.Conversion = &SSPIConfig{}
 )
 
-// jsonUnmarshalHandleDoubleEncode - due to a bug in xorm (see https://gitea.com/xorm/xorm/pulls/1957) - it's
+// JSONUnmarshalHandleDoubleEncode - due to a bug in xorm (see https://gitea.com/xorm/xorm/pulls/1957) - it's
 // possible that a Blob may be double encoded or gain an unwanted prefix of 0xff 0xfe.
-func jsonUnmarshalHandleDoubleEncode(bs []byte, v interface{}) error {
+func JSONUnmarshalHandleDoubleEncode(bs []byte, v interface{}) error {
 	json := jsoniter.ConfigCompatibleWithStandardLibrary
 	err := json.Unmarshal(bs, v)
 	if err != nil {
@@ -89,7 +89,7 @@ func jsonUnmarshalHandleDoubleEncode(bs []byte, v interface{}) error {
 			rs = append(rs, temp...)
 		}
 		if ok {
-			if rs[0] == 0xff && rs[1] == 0xfe {
+			if len(rs) > 1 && rs[0] == 0xff && rs[1] == 0xfe {
 				rs = rs[2:]
 			}
 			err = json.Unmarshal(rs, v)
@@ -108,7 +108,7 @@ type LDAPConfig struct {
 
 // FromDB fills up a LDAPConfig from serialized format.
 func (cfg *LDAPConfig) FromDB(bs []byte) error {
-	err := jsonUnmarshalHandleDoubleEncode(bs, &cfg)
+	err := JSONUnmarshalHandleDoubleEncode(bs, &cfg)
 	if err != nil {
 		return err
 	}
@@ -149,7 +149,7 @@ type SMTPConfig struct {
 
 // FromDB fills up an SMTPConfig from serialized format.
 func (cfg *SMTPConfig) FromDB(bs []byte) error {
-	return jsonUnmarshalHandleDoubleEncode(bs, cfg)
+	return JSONUnmarshalHandleDoubleEncode(bs, cfg)
 }
 
 // ToDB exports an SMTPConfig to a serialized format.
@@ -166,7 +166,7 @@ type PAMConfig struct {
 
 // FromDB fills up a PAMConfig from serialized format.
 func (cfg *PAMConfig) FromDB(bs []byte) error {
-	return jsonUnmarshalHandleDoubleEncode(bs, cfg)
+	return JSONUnmarshalHandleDoubleEncode(bs, cfg)
 }
 
 // ToDB exports a PAMConfig to a serialized format.
@@ -187,7 +187,7 @@ type OAuth2Config struct {
 
 // FromDB fills up an OAuth2Config from serialized format.
 func (cfg *OAuth2Config) FromDB(bs []byte) error {
-	return jsonUnmarshalHandleDoubleEncode(bs, cfg)
+	return JSONUnmarshalHandleDoubleEncode(bs, cfg)
 }
 
 // ToDB exports an SMTPConfig to a serialized format.
@@ -207,7 +207,7 @@ type SSPIConfig struct {
 
 // FromDB fills up an SSPIConfig from serialized format.
 func (cfg *SSPIConfig) FromDB(bs []byte) error {
-	return jsonUnmarshalHandleDoubleEncode(bs, cfg)
+	return JSONUnmarshalHandleDoubleEncode(bs, cfg)
 }
 
 // ToDB exports an SSPIConfig to a serialized format.

models/org.go

@@ -455,7 +455,7 @@ func GetUserOrgsList(user *User) ([]*MinimalOrg, error) {
 	groupByStr := groupByCols.String()
 	groupByStr = groupByStr[0 : len(groupByStr)-1]
 
-	sess.Select(groupByStr+", count(repo_id) as org_count").
+	sess.Select(groupByStr+", count(distinct repo_id) as org_count").
 		Table("user").
 		Join("INNER", "team", "`team`.org_id = `user`.id").
 		Join("INNER", "team_user", "`team`.id = `team_user`.team_id").

models/pull.go

@@ -502,6 +502,9 @@ func GetLatestPullRequestByHeadInfo(repoID int64, branch string) (*PullRequest,
 
 // GetPullRequestByIndex returns a pull request by the given index
 func GetPullRequestByIndex(repoID, index int64) (*PullRequest, error) {
+	if index < 1 {
+		return nil, ErrPullRequestNotExist{}
+	}
 	pr := &PullRequest{
 		BaseRepoID: repoID,
 		Index:      index,

models/pull_test.go

@@ -133,6 +133,10 @@ func TestGetPullRequestByIndex(t *testing.T) {
 	_, err = GetPullRequestByIndex(9223372036854775807, 9223372036854775807)
 	assert.Error(t, err)
 	assert.True(t, IsErrPullRequestNotExist(err))
+
+	_, err = GetPullRequestByIndex(1, 0)
+	assert.Error(t, err)
+	assert.True(t, IsErrPullRequestNotExist(err))
 }
 
 func TestGetPullRequestByID(t *testing.T) {

models/repo.go

@@ -1152,16 +1152,6 @@ func CreateRepository(ctx DBContext, doer, u *User, repo *Repository, overwriteO
 		return fmt.Errorf("recalculateAccesses: %v", err)
 	}
 
-	if u.Visibility == api.VisibleTypePublic && !repo.IsPrivate {
-		// Create/Remove git-daemon-export-ok for git-daemon...
-		daemonExportFile := path.Join(repo.RepoPath(), `git-daemon-export-ok`)
-		if f, err := os.Create(daemonExportFile); err != nil {
-			log.Error("Failed to create %s: %v", daemonExportFile, err)
-		} else {
-			f.Close()
-		}
-	}
-
 	if setting.Service.AutoWatchNewRepos {
 		if err = watchRepo(ctx.e, doer.ID, repo.ID, true); err != nil {
 			return fmt.Errorf("watchRepo: %v", err)
@@ -1175,6 +1165,46 @@ func CreateRepository(ctx DBContext, doer, u *User, repo *Repository, overwriteO
 	return nil
 }
 
+// CheckDaemonExportOK creates/removes git-daemon-export-ok for git-daemon...
+func (repo *Repository) CheckDaemonExportOK() error {
+	return repo.checkDaemonExportOK(x)
+}
+
+// CheckDaemonExportOKCtx creates/removes git-daemon-export-ok for git-daemon...
+func (repo *Repository) CheckDaemonExportOKCtx(ctx DBContext) error {
+	return repo.checkDaemonExportOK(ctx.e)
+}
+
+func (repo *Repository) checkDaemonExportOK(e Engine) error {
+	if err := repo.getOwner(e); err != nil {
+		return err
+	}
+
+	// Create/Remove git-daemon-export-ok for git-daemon...
+	daemonExportFile := path.Join(repo.RepoPath(), `git-daemon-export-ok`)
+
+	isExist, err := util.IsExist(daemonExportFile)
+	if err != nil {
+		log.Error("Unable to check if %s exists. Error: %v", daemonExportFile, err)
+		return err
+	}
+
+	isPublic := !repo.IsPrivate && repo.Owner.Visibility == api.VisibleTypePublic
+	if !isPublic && isExist {
+		if err = util.Remove(daemonExportFile); err != nil {
+			log.Error("Failed to remove %s: %v", daemonExportFile, err)
+		}
+	} else if isPublic && !isExist {
+		if f, err := os.Create(daemonExportFile); err != nil {
+			log.Error("Failed to create %s: %v", daemonExportFile, err)
+		} else {
+			f.Close()
+		}
+	}
+
+	return nil
+}
+
 func countRepositories(userID int64, private bool) int64 {
 	sess := x.Where("id > 0")
 
@@ -1217,6 +1247,12 @@ func IncrementRepoForkNum(ctx DBContext, repoID int64) error {
 	return err
 }
 
+// DecrementRepoForkNum decrement repository fork number
+func DecrementRepoForkNum(ctx DBContext, repoID int64) error {
+	_, err := ctx.e.Exec("UPDATE `repository` SET num_forks=num_forks-1 WHERE id=?", repoID)
+	return err
+}
+
 // ChangeRepositoryName changes all corresponding setting from old repository name to new one.
 func ChangeRepositoryName(doer *User, repo *Repository, newRepoName string) (err error) {
 	oldRepoName := repo.Name
@@ -1318,24 +1354,9 @@ func updateRepository(e Engine, repo *Repository, visibilityChanged bool) (err e
 		}
 
 		// Create/Remove git-daemon-export-ok for git-daemon...
-		daemonExportFile := path.Join(repo.RepoPath(), `git-daemon-export-ok`)
-		isExist, err := util.IsExist(daemonExportFile)
-		isPublic := !repo.IsPrivate && repo.Owner.Visibility == api.VisibleTypePublic
-		if err != nil {
-			log.Error("Unable to check if %s exists. Error: %v", daemonExportFile, err)
+		if err := repo.checkDaemonExportOK(e); err != nil {
 			return err
 		}
-		if !isPublic && isExist {
-			if err = util.Remove(daemonExportFile); err != nil {
-				log.Error("Failed to remove %s: %v", daemonExportFile, err)
-			}
-		} else if isPublic && !isExist {
-			if f, err := os.Create(daemonExportFile); err != nil {
-				log.Error("Failed to create %s: %v", daemonExportFile, err)
-			} else {
-				f.Close()
-			}
-		}
 
 		forkRepos, err := getRepositoriesByForkID(e, repo.ID)
 		if err != nil {

models/repo_collaboration.go

@@ -8,6 +8,7 @@ package models
 import (
 	"fmt"
 
+	"code.gitea.io/gitea/modules/log"
 	"code.gitea.io/gitea/modules/timeutil"
 
 	"xorm.io/builder"
@@ -83,16 +84,21 @@ func (repo *Repository) getCollaborators(e Engine, listOptions ListOptions) ([]*
 		return nil, fmt.Errorf("getCollaborations: %v", err)
 	}
 
-	collaborators := make([]*Collaborator, len(collaborations))
-	for i, c := range collaborations {
+	collaborators := make([]*Collaborator, 0, len(collaborations))
+	for _, c := range collaborations {
 		user, err := getUserByID(e, c.UserID)
 		if err != nil {
-			return nil, err
+			if IsErrUserNotExist(err) {
+				log.Warn("Inconsistent DB: User: %d is listed as collaborator of %-v but does not exist", c.UserID, repo)
+				user = NewGhostUser()
+			} else {
+				return nil, err
+			}
 		}
-		collaborators[i] = &Collaborator{
+		collaborators = append(collaborators, &Collaborator{
 			User:          user,
 			Collaboration: c,
-		}
+		})
 	}
 	return collaborators, nil
 }

models/repo_transfer.go

@@ -269,6 +269,14 @@ func TransferOwnership(doer *User, newOwnerName string, repo *Repository) (err e
 	// Dummy object.
 	collaboration := &Collaboration{RepoID: repo.ID}
 	for _, c := range collaborators {
+		if c.IsGhost() {
+			collaboration.ID = c.Collaboration.ID
+			if _, err := sess.Delete(collaboration); err != nil {
+				return fmt.Errorf("remove collaborator '%d': %v", c.ID, err)
+			}
+			collaboration.ID = 0
+		}
+
 		if c.ID != newOwner.ID {
 			isMember, err := isOrganizationMember(sess, newOwner.ID, c.ID)
 			if err != nil {
@@ -281,6 +289,7 @@ func TransferOwnership(doer *User, newOwnerName string, repo *Repository) (err e
 		if _, err := sess.Delete(collaboration); err != nil {
 			return fmt.Errorf("remove collaborator '%d': %v", c.ID, err)
 		}
+		collaboration.UserID = 0
 	}
 
 	// Remove old team-repository relations.

models/repo_unit.go

@@ -28,7 +28,7 @@ type UnitConfig struct{}
 
 // FromDB fills up a UnitConfig from serialized format.
 func (cfg *UnitConfig) FromDB(bs []byte) error {
-	return jsonUnmarshalHandleDoubleEncode(bs, &cfg)
+	return JSONUnmarshalHandleDoubleEncode(bs, &cfg)
 }
 
 // ToDB exports a UnitConfig to a serialized format.
@@ -44,7 +44,7 @@ type ExternalWikiConfig struct {
 
 // FromDB fills up a ExternalWikiConfig from serialized format.
 func (cfg *ExternalWikiConfig) FromDB(bs []byte) error {
-	return jsonUnmarshalHandleDoubleEncode(bs, &cfg)
+	return JSONUnmarshalHandleDoubleEncode(bs, &cfg)
 }
 
 // ToDB exports a ExternalWikiConfig to a serialized format.
@@ -62,7 +62,7 @@ type ExternalTrackerConfig struct {
 
 // FromDB fills up a ExternalTrackerConfig from serialized format.
 func (cfg *ExternalTrackerConfig) FromDB(bs []byte) error {
-	return jsonUnmarshalHandleDoubleEncode(bs, &cfg)
+	return JSONUnmarshalHandleDoubleEncode(bs, &cfg)
 }
 
 // ToDB exports a ExternalTrackerConfig to a serialized format.
@@ -80,7 +80,7 @@ type IssuesConfig struct {
 
 // FromDB fills up a IssuesConfig from serialized format.
 func (cfg *IssuesConfig) FromDB(bs []byte) error {
-	return jsonUnmarshalHandleDoubleEncode(bs, &cfg)
+	return JSONUnmarshalHandleDoubleEncode(bs, &cfg)
 }
 
 // ToDB exports a IssuesConfig to a serialized format.
@@ -104,7 +104,7 @@ type PullRequestsConfig struct {
 
 // FromDB fills up a PullRequestsConfig from serialized format.
 func (cfg *PullRequestsConfig) FromDB(bs []byte) error {
-	return jsonUnmarshalHandleDoubleEncode(bs, &cfg)
+	return JSONUnmarshalHandleDoubleEncode(bs, &cfg)
 }
 
 // ToDB exports a PullRequestsConfig to a serialized format.
@@ -219,3 +219,9 @@ func getUnitsByRepoID(e Engine, repoID int64) (units []*RepoUnit, err error) {
 
 	return units, nil
 }
+
+// UpdateRepoUnit updates the provided repo unit
+func UpdateRepoUnit(unit *RepoUnit) error {
+	_, err := x.ID(unit.ID).Update(unit)
+	return err
+}

models/review.go

@@ -434,7 +434,7 @@ func SubmitReview(doer *User, issue *Issue, reviewType ReviewType, content, comm
 	// try to remove team review request if need
 	if issue.Repo.Owner.IsOrganization() && (reviewType == ReviewTypeApprove || reviewType == ReviewTypeReject) {
 		teamReviewRequests := make([]*Review, 0, 10)
-		if err := sess.SQL("SELECT * FROM review WHERE reviewer_team_id > 0 AND type = ?", ReviewTypeRequest).Find(&teamReviewRequests); err != nil {
+		if err := sess.SQL("SELECT * FROM review WHERE issue_id = ? AND reviewer_team_id > 0 AND type = ?", issue.ID, ReviewTypeRequest).Find(&teamReviewRequests); err != nil {
 			return nil, nil, err
 		}
 

models/u2f.go

@@ -52,7 +52,7 @@ func (list U2FRegistrationList) ToRegistrations() []u2f.Registration {
 	for _, reg := range list {
 		r, err := reg.Parse()
 		if err != nil {
-			log.Fatal("parsing u2f registration: %v", err)
+			log.Error("parsing u2f registration: %v", err)
 			continue
 		}
 		regs = append(regs, *r)

models/u2f_test.go

@@ -5,6 +5,7 @@
 package models
 
 import (
+	"encoding/hex"
 	"testing"
 
 	"github.com/stretchr/testify/assert"
@@ -27,6 +28,7 @@ func TestGetU2FRegistrationsByUID(t *testing.T) {
 	assert.NoError(t, PrepareTestDatabase())
 
 	res, err := GetU2FRegistrationsByUID(1)
+
 	assert.NoError(t, err)
 	assert.Len(t, res, 1)
 	assert.Equal(t, "U2F Key", res[0].Name)
@@ -71,3 +73,27 @@ func TestDeleteRegistration(t *testing.T) {
 	assert.NoError(t, DeleteRegistration(reg))
 	AssertNotExistsBean(t, &U2FRegistration{ID: 1})
 }
+
+const validU2FRegistrationResponseHex = "0504b174bc49c7ca254b70d2e5c207cee9cf174820ebd77ea3c65508c26da51b657c1cc6b952f8621697936482da0a6d3d3826a59095daf6cd7c03e2e60385d2f6d9402a552dfdb7477ed65fd84133f86196010b2215b57da75d315b7b9e8fe2e3925a6019551bab61d16591659cbaf00b4950f7abfe6660e2e006f76868b772d70c253082013c3081e4a003020102020a47901280001155957352300a06082a8648ce3d0403023017311530130603550403130c476e756262792050696c6f74301e170d3132303831343138323933325a170d3133303831343138323933325a3031312f302d0603550403132650696c6f74476e756262792d302e342e312d34373930313238303030313135353935373335323059301306072a8648ce3d020106082a8648ce3d030107034200048d617e65c9508e64bcc5673ac82a6799da3c1446682c258c463fffdf58dfd2fa3e6c378b53d795c4a4dffb4199edd7862f23abaf0203b4b8911ba0569994e101300a06082a8648ce3d0403020347003044022060cdb6061e9c22262d1aac1d96d8c70829b2366531dda268832cb836bcd30dfa0220631b1459f09e6330055722c8d89b7f48883b9089b88d60d1d9795902b30410df304502201471899bcc3987e62e8202c9b39c33c19033f7340352dba80fcab017db9230e402210082677d673d891933ade6f617e5dbde2e247e70423fd5ad7804a6d3d3961ef871"
+
+func TestToRegistrations_SkipInvalidItemsWithoutCrashing(t *testing.T) {
+	regKeyRaw, _ := hex.DecodeString(validU2FRegistrationResponseHex)
+	regs := U2FRegistrationList{
+		&U2FRegistration{ID: 1},
+		&U2FRegistration{ID: 2, Name: "U2F Key", UserID: 2, Counter: 0, Raw: regKeyRaw, CreatedUnix: 946684800, UpdatedUnix: 946684800},
+	}
+
+	actual := regs.ToRegistrations()
+	assert.Len(t, actual, 1)
+}
+
+func TestToRegistrations(t *testing.T) {
+	regKeyRaw, _ := hex.DecodeString(validU2FRegistrationResponseHex)
+	regs := U2FRegistrationList{
+		&U2FRegistration{ID: 1, Name: "U2F Key", UserID: 1, Counter: 0, Raw: regKeyRaw, CreatedUnix: 946684800, UpdatedUnix: 946684800},
+		&U2FRegistration{ID: 2, Name: "U2F Key", UserID: 2, Counter: 0, Raw: regKeyRaw, CreatedUnix: 946684800, UpdatedUnix: 946684800},
+	}
+
+	actual := regs.ToRegistrations()
+	assert.Len(t, actual, 2)
+}

models/user.go

@@ -296,7 +296,7 @@ func (u *User) CanImportLocal() bool {
 // DashboardLink returns the user dashboard page link.
 func (u *User) DashboardLink() string {
 	if u.IsOrganization() {
-		return u.OrganisationLink() + "/dashboard/"
+		return u.OrganisationLink() + "/dashboard"
 	}
 	return setting.AppSubURL + "/"
 }
@@ -1062,9 +1062,9 @@ func checkDupEmail(e Engine, u *User) error {
 	return nil
 }
 
-// validateUser check if user is valide to insert / update into database
+// validateUser check if user is valid to insert / update into database
 func validateUser(u *User) error {
-	if !setting.Service.AllowedUserVisibilityModesSlice.IsAllowedVisibility(u.Visibility) {
+	if !setting.Service.AllowedUserVisibilityModesSlice.IsAllowedVisibility(u.Visibility) && !u.IsOrganization() {
 		return fmt.Errorf("visibility Mode not allowed: %s", u.Visibility.String())
 	}
 

models/user_heatmap_test.go

@@ -7,6 +7,9 @@ package models
 import (
 	"fmt"
 	"testing"
+	"time"
+
+	"code.gitea.io/gitea/modules/timeutil"
 
 	jsoniter "github.com/json-iterator/go"
 	"github.com/stretchr/testify/assert"
@@ -37,6 +40,10 @@ func TestGetUserHeatmapDataByUser(t *testing.T) {
 	// Prepare
 	assert.NoError(t, PrepareTestDatabase())
 
+	// Mock time
+	timeutil.Set(time.Date(2021, 1, 1, 0, 0, 0, 0, time.UTC))
+	defer timeutil.Unset()
+
 	for i, tc := range testCases {
 		user := AssertExistsAndLoadBean(t, &User{ID: tc.userID}).(*User)
 

modules/context/context.go

@@ -587,6 +587,17 @@ func GetContext(req *http.Request) *Context {
 	return req.Context().Value(contextKey).(*Context)
 }
 
+// GetContextUser returns context user
+func GetContextUser(req *http.Request) *models.User {
+	if apiContext, ok := req.Context().Value(apiContextKey).(*APIContext); ok {
+		return apiContext.User
+	}
+	if ctx, ok := req.Context().Value(contextKey).(*Context); ok {
+		return ctx.User
+	}
+	return nil
+}
+
 // SignedUserName returns signed user's name via context
 func SignedUserName(req *http.Request) string {
 	if middleware.IsInternalPath(req) {

modules/context/repo.go

@@ -695,7 +695,7 @@ func getRefName(ctx *Context, pathType RepoRefType) string {
 		}
 		// For legacy and API support only full commit sha
 		parts := strings.Split(path, "/")
-		if len(parts) > 0 && len(parts[0]) == 40 {
+		if len(parts) > 1 && len(parts[0]) == 40 {
 			ctx.Repo.TreePath = strings.Join(parts[1:], "/")
 			return parts[0]
 		}

modules/convert/pull.go

@@ -17,7 +17,7 @@ import (
 // ToAPIPullRequest assumes following fields have been assigned with valid values:
 // Required - Issue
 // Optional - Merger
-func ToAPIPullRequest(pr *models.PullRequest) *api.PullRequest {
+func ToAPIPullRequest(pr *models.PullRequest, doer *models.User) *api.PullRequest {
 	var (
 		baseBranch *git.Branch
 		headBranch *git.Branch
@@ -41,6 +41,12 @@ func ToAPIPullRequest(pr *models.PullRequest) *api.PullRequest {
 		return nil
 	}
 
+	perm, err := models.GetUserRepoPermission(pr.BaseRepo, doer)
+	if err != nil {
+		log.Error("GetUserRepoPermission[%d]: %v", pr.BaseRepoID, err)
+		perm.AccessMode = models.AccessModeNone
+	}
+
 	apiPullRequest := &api.PullRequest{
 		ID:        pr.ID,
 		URL:       pr.Issue.HTMLURL(),
@@ -68,7 +74,7 @@ func ToAPIPullRequest(pr *models.PullRequest) *api.PullRequest {
 			Name:       pr.BaseBranch,
 			Ref:        pr.BaseBranch,
 			RepoID:     pr.BaseRepoID,
-			Repository: ToRepo(pr.BaseRepo, models.AccessModeNone),
+			Repository: ToRepo(pr.BaseRepo, perm.AccessMode),
 		},
 		Head: &api.PRBranchInfo{
 			Name:   pr.HeadBranch,
@@ -96,8 +102,14 @@ func ToAPIPullRequest(pr *models.PullRequest) *api.PullRequest {
 	}
 
 	if pr.HeadRepo != nil {
+		perm, err := models.GetUserRepoPermission(pr.HeadRepo, doer)
+		if err != nil {
+			log.Error("GetUserRepoPermission[%d]: %v", pr.HeadRepoID, err)
+			perm.AccessMode = models.AccessModeNone
+		}
+
 		apiPullRequest.Head.RepoID = pr.HeadRepo.ID
-		apiPullRequest.Head.Repository = ToRepo(pr.HeadRepo, models.AccessModeNone)
+		apiPullRequest.Head.Repository = ToRepo(pr.HeadRepo, perm.AccessMode)
 
 		headGitRepo, err := git.OpenRepository(pr.HeadRepo.RepoPath())
 		if err != nil {

modules/convert/pull_test.go

@@ -20,14 +20,14 @@ func TestPullRequest_APIFormat(t *testing.T) {
 	pr := models.AssertExistsAndLoadBean(t, &models.PullRequest{ID: 1}).(*models.PullRequest)
 	assert.NoError(t, pr.LoadAttributes())
 	assert.NoError(t, pr.LoadIssue())
-	apiPullRequest := ToAPIPullRequest(pr)
+	apiPullRequest := ToAPIPullRequest(pr, nil)
 	assert.NotNil(t, apiPullRequest)
 	assert.EqualValues(t, &structs.PRBranchInfo{
 		Name:       "branch1",
 		Ref:        "refs/pull/2/head",
 		Sha:        "4a357436d925b5c974181ff12a994538ddc5a269",
 		RepoID:     1,
-		Repository: ToRepo(headRepo, models.AccessModeNone),
+		Repository: ToRepo(headRepo, models.AccessModeRead),
 	}, apiPullRequest.Head)
 
 	//withOut HeadRepo
@@ -37,7 +37,7 @@ func TestPullRequest_APIFormat(t *testing.T) {
 	// simulate fork deletion
 	pr.HeadRepo = nil
 	pr.HeadRepoID = 100000
-	apiPullRequest = ToAPIPullRequest(pr)
+	apiPullRequest = ToAPIPullRequest(pr, nil)
 	assert.NotNil(t, apiPullRequest)
 	assert.Nil(t, apiPullRequest.Head.Repository)
 	assert.EqualValues(t, -1, apiPullRequest.Head.RepoID)

modules/csv/csv.go

@@ -32,6 +32,9 @@ func CreateReaderAndGuessDelimiter(rd io.Reader) (*stdcsv.Reader, error) {
 	var data = make([]byte, 1e4)
 	size, err := rd.Read(data)
 	if err != nil {
+		if err == io.EOF {
+			return CreateReader(bytes.NewReader([]byte{}), rune(',')), nil
+		}
 		return nil, err
 	}
 

modules/doctor/dbconsistency.go

@@ -13,6 +13,64 @@ import (
 	"code.gitea.io/gitea/modules/setting"
 )
 
+type consistencyCheck struct {
+	Name         string
+	Counter      func() (int64, error)
+	Fixer        func() (int64, error)
+	FixedMessage string
+}
+
+func (c *consistencyCheck) Run(logger log.Logger, autofix bool) error {
+	count, err := c.Counter()
+	if err != nil {
+		logger.Critical("Error: %v whilst counting %s", err, c.Name)
+		return err
+	}
+	if count > 0 {
+		if autofix {
+			var fixed int64
+			if fixed, err = c.Fixer(); err != nil {
+				logger.Critical("Error: %v whilst fixing %s", err, c.Name)
+				return err
+			}
+
+			prompt := "Deleted"
+			if c.FixedMessage != "" {
+				prompt = c.FixedMessage
+			}
+
+			if fixed < 0 {
+				logger.Info(prompt+" %d %s", count, c.Name)
+			} else {
+				logger.Info(prompt+" %d/%d %s", fixed, count, c.Name)
+			}
+		} else {
+			logger.Warn("Found %d %s", count, c.Name)
+		}
+	}
+	return nil
+}
+
+func asFixer(fn func() error) func() (int64, error) {
+	return func() (int64, error) {
+		err := fn()
+		return -1, err
+	}
+}
+
+func genericOrphanCheck(name, subject, refobject, joincond string) consistencyCheck {
+	return consistencyCheck{
+		Name: name,
+		Counter: func() (int64, error) {
+			return models.CountOrphanedObjects(subject, refobject, joincond)
+		},
+		Fixer: func() (int64, error) {
+			err := models.DeleteOrphanedObjects(subject, refobject, joincond)
+			return -1, err
+		},
+	}
+}
+
 func checkDBConsistency(logger log.Logger, autofix bool) error {
 	// make sure DB version is uptodate
 	if err := models.NewEngine(context.Background(), migrations.EnsureUpToDate); err != nil {
@@ -20,246 +78,103 @@ func checkDBConsistency(logger log.Logger, autofix bool) error {
 		return err
 	}
 
-	// find labels without existing repo or org
-	count, err := models.CountOrphanedLabels()
-	if err != nil {
-		logger.Critical("Error: %v whilst counting orphaned labels", err)
-		return err
-	}
-	if count > 0 {
-		if autofix {
-			if err = models.DeleteOrphanedLabels(); err != nil {
-				logger.Critical("Error: %v whilst deleting orphaned labels", err)
-				return err
-			}
-			logger.Info("%d labels without existing repository/organisation deleted", count)
-		} else {
-			logger.Warn("%d labels without existing repository/organisation", count)
-		}
-	}
-
-	// find IssueLabels without existing label
-	count, err = models.CountOrphanedIssueLabels()
-	if err != nil {
-		logger.Critical("Error: %v whilst counting orphaned issue_labels", err)
-		return err
-	}
-	if count > 0 {
-		if autofix {
-			if err = models.DeleteOrphanedIssueLabels(); err != nil {
-				logger.Critical("Error: %v whilst deleting orphaned issue_labels", err)
-				return err
-			}
-			logger.Info("%d issue_labels without existing label deleted", count)
-		} else {
-			logger.Warn("%d issue_labels without existing label", count)
-		}
-	}
-
-	// find issues without existing repository
-	count, err = models.CountOrphanedIssues()
-	if err != nil {
-		logger.Critical("Error: %v whilst counting orphaned issues", err)
-		return err
-	}
-	if count > 0 {
-		if autofix {
-			if err = models.DeleteOrphanedIssues(); err != nil {
-				logger.Critical("Error: %v whilst deleting orphaned issues", err)
-				return err
-			}
-			logger.Info("%d issues without existing repository deleted", count)
-		} else {
-			logger.Warn("%d issues without existing repository", count)
-		}
-	}
-
-	// find pulls without existing issues
-	count, err = models.CountOrphanedObjects("pull_request", "issue", "pull_request.issue_id=issue.id")
-	if err != nil {
-		logger.Critical("Error: %v whilst counting orphaned objects", err)
-		return err
-	}
-	if count > 0 {
-		if autofix {
-			if err = models.DeleteOrphanedObjects("pull_request", "issue", "pull_request.issue_id=issue.id"); err != nil {
-				logger.Critical("Error: %v whilst deleting orphaned objects", err)
-				return err
-			}
-			logger.Info("%d pull requests without existing issue deleted", count)
-		} else {
-			logger.Warn("%d pull requests without existing issue", count)
-		}
-	}
-
-	// find tracked times without existing issues/pulls
-	count, err = models.CountOrphanedObjects("tracked_time", "issue", "tracked_time.issue_id=issue.id")
-	if err != nil {
-		logger.Critical("Error: %v whilst counting orphaned objects", err)
-		return err
-	}
-	if count > 0 {
-		if autofix {
-			if err = models.DeleteOrphanedObjects("tracked_time", "issue", "tracked_time.issue_id=issue.id"); err != nil {
-				logger.Critical("Error: %v whilst deleting orphaned objects", err)
-				return err
-			}
-			logger.Info("%d tracked times without existing issue deleted", count)
-		} else {
-			logger.Warn("%d tracked times without existing issue", count)
-		}
-	}
-
-	// find null archived repositories
-	count, err = models.CountNullArchivedRepository()
-	if err != nil {
-		logger.Critical("Error: %v whilst counting null archived repositories", err)
-		return err
-	}
-	if count > 0 {
-		if autofix {
-			updatedCount, err := models.FixNullArchivedRepository()
-			if err != nil {
-				logger.Critical("Error: %v whilst fixing null archived repositories", err)
-				return err
-			}
-			logger.Info("%d repositories with null is_archived updated", updatedCount)
-		} else {
-			logger.Warn("%d repositories with null is_archived", count)
-		}
-	}
-
-	// find label comments with empty labels
-	count, err = models.CountCommentTypeLabelWithEmptyLabel()
-	if err != nil {
-		logger.Critical("Error: %v whilst counting label comments with empty labels", err)
-		return err
-	}
-	if count > 0 {
-		if autofix {
-			updatedCount, err := models.FixCommentTypeLabelWithEmptyLabel()
-			if err != nil {
-				logger.Critical("Error: %v whilst removing label comments with empty labels", err)
-				return err
-			}
-			logger.Info("%d label comments with empty labels removed", updatedCount)
-		} else {
-			logger.Warn("%d label comments with empty labels", count)
-		}
-	}
-
-	// find label comments with labels from outside the repository
-	count, err = models.CountCommentTypeLabelWithOutsideLabels()
-	if err != nil {
-		logger.Critical("Error: %v whilst counting label comments with outside labels", err)
-		return err
-	}
-	if count > 0 {
-		if autofix {
-			updatedCount, err := models.FixCommentTypeLabelWithOutsideLabels()
-			if err != nil {
-				logger.Critical("Error: %v whilst removing label comments with outside labels", err)
-				return err
-			}
-			log.Info("%d label comments with outside labels removed", updatedCount)
-		} else {
-			log.Warn("%d label comments with outside labels", count)
-		}
-	}
-
-	// find issue_label with labels from outside the repository
-	count, err = models.CountIssueLabelWithOutsideLabels()
-	if err != nil {
-		logger.Critical("Error: %v whilst counting issue_labels from outside the repository or organisation", err)
-		return err
-	}
-	if count > 0 {
-		if autofix {
-			updatedCount, err := models.FixIssueLabelWithOutsideLabels()
-			if err != nil {
-				logger.Critical("Error: %v whilst removing issue_labels from outside the repository or organisation", err)
-				return err
-			}
-			logger.Info("%d issue_labels from outside the repository or organisation removed", updatedCount)
-		} else {
-			logger.Warn("%d issue_labels from outside the repository or organisation", count)
-		}
+	consistencyChecks := []consistencyCheck{
+		{
+			// find labels without existing repo or org
+			Name:    "Orphaned Labels without existing repository or organisation",
+			Counter: models.CountOrphanedLabels,
+			Fixer:   asFixer(models.DeleteOrphanedLabels),
+		},
+		{
+			// find IssueLabels without existing label
+			Name:    "Orphaned Issue Labels without existing label",
+			Counter: models.CountOrphanedIssueLabels,
+			Fixer:   asFixer(models.DeleteOrphanedIssueLabels),
+		},
+		{
+			// find issues without existing repository
+			Name:    "Orphaned Issues without existing repository",
+			Counter: models.CountOrphanedIssues,
+			Fixer:   asFixer(models.DeleteOrphanedIssues),
+		},
+		// find releases without existing repository
+		genericOrphanCheck("Orphaned Releases without existing repository",
+			"release", "repository", "release.repo_id=repository.id"),
+		// find pulls without existing issues
+		genericOrphanCheck("Orphaned PullRequests without existing issue",
+			"pull_request", "issue", "pull_request.issue_id=issue.id"),
+		// find tracked times without existing issues/pulls
+		genericOrphanCheck("Orphaned TrackedTimes without existing issue",
+			"tracked_time", "issue", "tracked_time.issue_id=issue.id"),
+		// find null archived repositories
+		{
+			Name:         "Repositories with is_archived IS NULL",
+			Counter:      models.CountNullArchivedRepository,
+			Fixer:        models.FixNullArchivedRepository,
+			FixedMessage: "Fixed",
+		},
+		// find label comments with empty labels
+		{
+			Name:         "Label comments with empty labels",
+			Counter:      models.CountCommentTypeLabelWithEmptyLabel,
+			Fixer:        models.FixCommentTypeLabelWithEmptyLabel,
+			FixedMessage: "Fixed",
+		},
+		// find label comments with labels from outside the repository
+		{
+			Name:         "Label comments with labels from outside the repository",
+			Counter:      models.CountCommentTypeLabelWithOutsideLabels,
+			Fixer:        models.FixCommentTypeLabelWithOutsideLabels,
+			FixedMessage: "Removed",
+		},
+		// find issue_label with labels from outside the repository
+		{
+			Name:         "IssueLabels with Labels from outside the repository",
+			Counter:      models.CountIssueLabelWithOutsideLabels,
+			Fixer:        models.FixIssueLabelWithOutsideLabels,
+			FixedMessage: "Removed",
+		},
 	}
 
 	// TODO: function to recalc all counters
 
 	if setting.Database.UsePostgreSQL {
-		count, err = models.CountBadSequences()
-		if err != nil {
-			logger.Critical("Error: %v whilst checking sequence values", err)
+		consistencyChecks = append(consistencyChecks, consistencyCheck{
+			Name:         "Sequence values",
+			Counter:      models.CountBadSequences,
+			Fixer:        asFixer(models.FixBadSequences),
+			FixedMessage: "Updated",
+		})
+	}
+
+	consistencyChecks = append(consistencyChecks,
+		// find protected branches without existing repository
+		genericOrphanCheck("Protected Branches without existing repository",
+			"protected_branch", "repository", "protected_branch.repo_id=repository.id"),
+		// find deleted branches without existing repository
+		genericOrphanCheck("Deleted Branches without existing repository",
+			"deleted_branch", "repository", "deleted_branch.repo_id=repository.id"),
+		// find LFS locks without existing repository
+		genericOrphanCheck("LFS locks without existing repository",
+			"lfs_lock", "repository", "lfs_lock.repo_id=repository.id"),
+		// find collaborations without users
+		genericOrphanCheck("Collaborations without existing user",
+			"collaboration", "user", "collaboration.user_id=user.id"),
+		// find collaborations without repository
+		genericOrphanCheck("Collaborations without existing repository",
+			"collaboration", "repository", "collaboration.repo_id=repository.id"),
+		// find access without users
+		genericOrphanCheck("Access entries without existing user",
+			"access", "user", "access.user_id=user.id"),
+		// find access without repository
+		genericOrphanCheck("Access entries without existing repository",
+			"access", "repository", "access.repo_id=repository.id"),
+	)
+
+	for _, c := range consistencyChecks {
+		if err := c.Run(logger, autofix); err != nil {
 			return err
 		}
-		if count > 0 {
-			if autofix {
-				err := models.FixBadSequences()
-				if err != nil {
-					logger.Critical("Error: %v whilst attempting to fix sequences", err)
-					return err
-				}
-				logger.Info("%d sequences updated", count)
-			} else {
-				logger.Warn("%d sequences with incorrect values", count)
-			}
-		}
-	}
-
-	// find protected branches without existing repository
-	count, err = models.CountOrphanedObjects("protected_branch", "repository", "protected_branch.repo_id=repository.id")
-	if err != nil {
-		logger.Critical("Error: %v whilst counting orphaned objects", err)
-		return err
-	}
-	if count > 0 {
-		if autofix {
-			if err = models.DeleteOrphanedObjects("protected_branch", "repository", "protected_branch.repo_id=repository.id"); err != nil {
-				logger.Critical("Error: %v whilst deleting orphaned objects", err)
-				return err
-			}
-			logger.Info("%d protected branches without existing repository deleted", count)
-		} else {
-			logger.Warn("%d protected branches without existing repository", count)
-		}
-	}
-
-	// find deleted branches without existing repository
-	count, err = models.CountOrphanedObjects("deleted_branch", "repository", "deleted_branch.repo_id=repository.id")
-	if err != nil {
-		logger.Critical("Error: %v whilst counting orphaned objects", err)
-		return err
-	}
-	if count > 0 {
-		if autofix {
-			if err = models.DeleteOrphanedObjects("deleted_branch", "repository", "deleted_branch.repo_id=repository.id"); err != nil {
-				logger.Critical("Error: %v whilst deleting orphaned objects", err)
-				return err
-			}
-			logger.Info("%d deleted branches without existing repository deleted", count)
-		} else {
-			logger.Warn("%d deleted branches without existing repository", count)
-		}
-	}
-
-	// find LFS locks without existing repository
-	count, err = models.CountOrphanedObjects("lfs_lock", "repository", "lfs_lock.repo_id=repository.id")
-	if err != nil {
-		logger.Critical("Error: %v whilst counting orphaned objects", err)
-		return err
-	}
-	if count > 0 {
-		if autofix {
-			if err = models.DeleteOrphanedObjects("lfs_lock", "repository", "lfs_lock.repo_id=repository.id"); err != nil {
-				logger.Critical("Error: %v whilst deleting orphaned objects", err)
-				return err
-			}
-			logger.Info("%d LFS locks without existing repository deleted", count)
-		} else {
-			logger.Warn("%d LFS locks without existing repository", count)
-		}
 	}
 
 	return nil

modules/doctor/fix16961.go

@@ -0,0 +1,317 @@
+// Copyright 2021 The Gitea Authors. All rights reserved.
+// Use of this source code is governed by a MIT-style
+// license that can be found in the LICENSE file.
+
+package doctor
+
+import (
+	"bytes"
+	"fmt"
+
+	"code.gitea.io/gitea/models"
+	"code.gitea.io/gitea/modules/log"
+	"code.gitea.io/gitea/modules/timeutil"
+	"xorm.io/builder"
+)
+
+// #16831 revealed that the dump command that was broken in 1.14.3-1.14.6 and 1.15.0 (#15885).
+// This led to repo_unit and login_source cfg not being converted to JSON in the dump
+// Unfortunately although it was hoped that there were only a few users affected it
+// appears that many users are affected.
+
+// We therefore need to provide a doctor command to fix this repeated issue #16961
+
+func parseBool16961(bs []byte) (bool, error) {
+	if bytes.EqualFold(bs, []byte("%!s(bool=false)")) {
+		return false, nil
+	}
+
+	if bytes.EqualFold(bs, []byte("%!s(bool=true)")) {
+		return true, nil
+	}
+
+	return false, fmt.Errorf("unexpected bool format: %s", string(bs))
+}
+
+func fixUnitConfig16961(bs []byte, cfg *models.UnitConfig) (fixed bool, err error) {
+	err = models.JSONUnmarshalHandleDoubleEncode(bs, &cfg)
+	if err == nil {
+		return
+	}
+
+	// Handle #16961
+	if string(bs) != "&{}" && len(bs) != 0 {
+		return
+	}
+
+	return true, nil
+}
+
+func fixExternalWikiConfig16961(bs []byte, cfg *models.ExternalWikiConfig) (fixed bool, err error) {
+	err = models.JSONUnmarshalHandleDoubleEncode(bs, &cfg)
+	if err == nil {
+		return
+	}
+
+	if len(bs) < 3 {
+		return
+	}
+	if bs[0] != '&' || bs[1] != '{' || bs[len(bs)-1] != '}' {
+		return
+	}
+	cfg.ExternalWikiURL = string(bs[2 : len(bs)-1])
+	return true, nil
+}
+
+func fixExternalTrackerConfig16961(bs []byte, cfg *models.ExternalTrackerConfig) (fixed bool, err error) {
+	err = models.JSONUnmarshalHandleDoubleEncode(bs, &cfg)
+	if err == nil {
+		return
+	}
+	// Handle #16961
+	if len(bs) < 3 {
+		return
+	}
+
+	if bs[0] != '&' || bs[1] != '{' || bs[len(bs)-1] != '}' {
+		return
+	}
+
+	parts := bytes.Split(bs[2:len(bs)-1], []byte{' '})
+	if len(parts) != 3 {
+		return
+	}
+
+	cfg.ExternalTrackerURL = string(bytes.Join(parts[:len(parts)-2], []byte{' '}))
+	cfg.ExternalTrackerFormat = string(parts[len(parts)-2])
+	cfg.ExternalTrackerStyle = string(parts[len(parts)-1])
+	return true, nil
+}
+
+func fixPullRequestsConfig16961(bs []byte, cfg *models.PullRequestsConfig) (fixed bool, err error) {
+	err = models.JSONUnmarshalHandleDoubleEncode(bs, &cfg)
+	if err == nil {
+		return
+	}
+
+	// Handle #16961
+	if len(bs) < 3 {
+		return
+	}
+
+	if bs[0] != '&' || bs[1] != '{' || bs[len(bs)-1] != '}' {
+		return
+	}
+
+	// PullRequestsConfig was the following in 1.14
+	// type PullRequestsConfig struct {
+	// 	IgnoreWhitespaceConflicts bool
+	// 	AllowMerge                bool
+	// 	AllowRebase               bool
+	// 	AllowRebaseMerge          bool
+	// 	AllowSquash               bool
+	// 	AllowManualMerge          bool
+	// 	AutodetectManualMerge     bool
+	// }
+	//
+	// 1.15 added in addition:
+	// DefaultDeleteBranchAfterMerge bool
+	// DefaultMergeStyle             MergeStyle
+	parts := bytes.Split(bs[2:len(bs)-1], []byte{' '})
+	if len(parts) < 7 {
+		return
+	}
+
+	var parseErr error
+	cfg.IgnoreWhitespaceConflicts, parseErr = parseBool16961(parts[0])
+	if parseErr != nil {
+		return
+	}
+	cfg.AllowMerge, parseErr = parseBool16961(parts[1])
+	if parseErr != nil {
+		return
+	}
+	cfg.AllowRebase, parseErr = parseBool16961(parts[2])
+	if parseErr != nil {
+		return
+	}
+	cfg.AllowRebaseMerge, parseErr = parseBool16961(parts[3])
+	if parseErr != nil {
+		return
+	}
+	cfg.AllowSquash, parseErr = parseBool16961(parts[4])
+	if parseErr != nil {
+		return
+	}
+	cfg.AllowManualMerge, parseErr = parseBool16961(parts[5])
+	if parseErr != nil {
+		return
+	}
+	cfg.AutodetectManualMerge, parseErr = parseBool16961(parts[6])
+	if parseErr != nil {
+		return
+	}
+
+	// 1.14 unit
+	if len(parts) == 7 {
+		return true, nil
+	}
+
+	if len(parts) < 9 {
+		return
+	}
+
+	cfg.DefaultDeleteBranchAfterMerge, parseErr = parseBool16961(parts[7])
+	if parseErr != nil {
+		return
+	}
+
+	cfg.DefaultMergeStyle = models.MergeStyle(string(bytes.Join(parts[8:], []byte{' '})))
+	return true, nil
+}
+
+func fixIssuesConfig16961(bs []byte, cfg *models.IssuesConfig) (fixed bool, err error) {
+	err = models.JSONUnmarshalHandleDoubleEncode(bs, &cfg)
+	if err == nil {
+		return
+	}
+
+	// Handle #16961
+	if len(bs) < 3 {
+		return
+	}
+
+	if bs[0] != '&' || bs[1] != '{' || bs[len(bs)-1] != '}' {
+		return
+	}
+
+	parts := bytes.Split(bs[2:len(bs)-1], []byte{' '})
+	if len(parts) != 3 {
+		return
+	}
+	var parseErr error
+	cfg.EnableTimetracker, parseErr = parseBool16961(parts[0])
+	if parseErr != nil {
+		return
+	}
+	cfg.AllowOnlyContributorsToTrackTime, parseErr = parseBool16961(parts[1])
+	if parseErr != nil {
+		return
+	}
+	cfg.EnableDependencies, parseErr = parseBool16961(parts[2])
+	if parseErr != nil {
+		return
+	}
+	return true, nil
+}
+
+func fixBrokenRepoUnit16961(repoUnit *models.RepoUnit, bs []byte) (fixed bool, err error) {
+	// Shortcut empty or null values
+	if len(bs) == 0 {
+		return false, nil
+	}
+
+	switch models.UnitType(repoUnit.Type) {
+	case models.UnitTypeCode, models.UnitTypeReleases, models.UnitTypeWiki, models.UnitTypeProjects:
+		cfg := &models.UnitConfig{}
+		repoUnit.Config = cfg
+		if fixed, err := fixUnitConfig16961(bs, cfg); !fixed {
+			return false, err
+		}
+	case models.UnitTypeExternalWiki:
+		cfg := &models.ExternalWikiConfig{}
+		repoUnit.Config = cfg
+
+		if fixed, err := fixExternalWikiConfig16961(bs, cfg); !fixed {
+			return false, err
+		}
+	case models.UnitTypeExternalTracker:
+		cfg := &models.ExternalTrackerConfig{}
+		repoUnit.Config = cfg
+		if fixed, err := fixExternalTrackerConfig16961(bs, cfg); !fixed {
+			return false, err
+		}
+	case models.UnitTypePullRequests:
+		cfg := &models.PullRequestsConfig{}
+		repoUnit.Config = cfg
+
+		if fixed, err := fixPullRequestsConfig16961(bs, cfg); !fixed {
+			return false, err
+		}
+	case models.UnitTypeIssues:
+		cfg := &models.IssuesConfig{}
+		repoUnit.Config = cfg
+		if fixed, err := fixIssuesConfig16961(bs, cfg); !fixed {
+			return false, err
+		}
+	default:
+		panic(fmt.Sprintf("unrecognized repo unit type: %v", repoUnit.Type))
+	}
+	return true, nil
+}
+
+func fixBrokenRepoUnits16961(logger log.Logger, autofix bool) error {
+	// RepoUnit describes all units of a repository
+	type RepoUnit struct {
+		ID          int64
+		RepoID      int64
+		Type        models.UnitType
+		Config      []byte
+		CreatedUnix timeutil.TimeStamp `xorm:"INDEX CREATED"`
+	}
+
+	count := 0
+
+	err := models.Iterate(
+		models.DefaultDBContext(),
+		new(RepoUnit),
+		builder.Gt{
+			"id": 0,
+		},
+		func(idx int, bean interface{}) error {
+			unit := bean.(*RepoUnit)
+
+			bs := unit.Config
+			repoUnit := &models.RepoUnit{
+				ID:          unit.ID,
+				RepoID:      unit.RepoID,
+				Type:        unit.Type,
+				CreatedUnix: unit.CreatedUnix,
+			}
+
+			if fixed, err := fixBrokenRepoUnit16961(repoUnit, bs); !fixed {
+				return err
+			}
+
+			count++
+			if !autofix {
+				return nil
+			}
+
+			return models.UpdateRepoUnit(repoUnit)
+		},
+	)
+
+	if err != nil {
+		logger.Critical("Unable to iterate acrosss repounits to fix the broken units: Error %v", err)
+		return err
+	}
+
+	if !autofix {
+		logger.Warn("Found %d broken repo_units", count)
+		return nil
+	}
+	logger.Info("Fixed %d broken repo_units", count)
+
+	return nil
+}
+
+func init() {
+	Register(&Check{
+		Title:     "Check for incorrectly dumped repo_units (See #16961)",
+		Name:      "fix-broken-repo-units",
+		IsDefault: false,
+		Run:       fixBrokenRepoUnits16961,
+		Priority:  7,
+	})
+}

modules/doctor/fix16961_test.go

@@ -0,0 +1,271 @@
+// Copyright 2021 The Gitea Authors. All rights reserved.
+// Use of this source code is governed by a MIT-style
+// license that can be found in the LICENSE file.
+
+package doctor
+
+import (
+	"testing"
+
+	"code.gitea.io/gitea/models"
+	"github.com/stretchr/testify/assert"
+)
+
+func Test_fixUnitConfig_16961(t *testing.T) {
+	tests := []struct {
+		name      string
+		bs        string
+		wantFixed bool
+		wantErr   bool
+	}{
+		{
+			name:      "empty",
+			bs:        "",
+			wantFixed: true,
+			wantErr:   false,
+		},
+		{
+			name:      "normal: {}",
+			bs:        "{}",
+			wantFixed: false,
+			wantErr:   false,
+		},
+		{
+			name:      "broken but fixable: &{}",
+			bs:        "&{}",
+			wantFixed: true,
+			wantErr:   false,
+		},
+		{
+			name:      "broken but unfixable: &{asdasd}",
+			bs:        "&{asdasd}",
+			wantFixed: false,
+			wantErr:   true,
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			gotFixed, err := fixUnitConfig16961([]byte(tt.bs), &models.UnitConfig{})
+			if (err != nil) != tt.wantErr {
+				t.Errorf("fixUnitConfig_16961() error = %v, wantErr %v", err, tt.wantErr)
+				return
+			}
+			if gotFixed != tt.wantFixed {
+				t.Errorf("fixUnitConfig_16961() = %v, want %v", gotFixed, tt.wantFixed)
+			}
+		})
+	}
+}
+
+func Test_fixExternalWikiConfig_16961(t *testing.T) {
+	tests := []struct {
+		name      string
+		bs        string
+		expected  string
+		wantFixed bool
+		wantErr   bool
+	}{
+		{
+			name:      "normal: {\"ExternalWikiURL\":\"http://someurl\"}",
+			bs:        "{\"ExternalWikiURL\":\"http://someurl\"}",
+			expected:  "http://someurl",
+			wantFixed: false,
+			wantErr:   false,
+		},
+		{
+			name:      "broken: &{http://someurl}",
+			bs:        "&{http://someurl}",
+			expected:  "http://someurl",
+			wantFixed: true,
+			wantErr:   false,
+		},
+		{
+			name:      "broken but unfixable: http://someurl",
+			bs:        "http://someurl",
+			wantFixed: false,
+			wantErr:   true,
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			cfg := &models.ExternalWikiConfig{}
+			gotFixed, err := fixExternalWikiConfig16961([]byte(tt.bs), cfg)
+			if (err != nil) != tt.wantErr {
+				t.Errorf("fixExternalWikiConfig_16961() error = %v, wantErr %v", err, tt.wantErr)
+				return
+			}
+			if gotFixed != tt.wantFixed {
+				t.Errorf("fixExternalWikiConfig_16961() = %v, want %v", gotFixed, tt.wantFixed)
+			}
+			if cfg.ExternalWikiURL != tt.expected {
+				t.Errorf("fixExternalWikiConfig_16961().ExternalWikiURL = %v, want %v", cfg.ExternalWikiURL, tt.expected)
+			}
+		})
+	}
+}
+
+func Test_fixExternalTrackerConfig_16961(t *testing.T) {
+	tests := []struct {
+		name      string
+		bs        string
+		expected  models.ExternalTrackerConfig
+		wantFixed bool
+		wantErr   bool
+	}{
+		{
+			name: "normal",
+			bs:   `{"ExternalTrackerURL":"a","ExternalTrackerFormat":"b","ExternalTrackerStyle":"c"}`,
+			expected: models.ExternalTrackerConfig{
+				ExternalTrackerURL:    "a",
+				ExternalTrackerFormat: "b",
+				ExternalTrackerStyle:  "c",
+			},
+			wantFixed: false,
+			wantErr:   false,
+		},
+		{
+			name: "broken",
+			bs:   "&{a b c}",
+			expected: models.ExternalTrackerConfig{
+				ExternalTrackerURL:    "a",
+				ExternalTrackerFormat: "b",
+				ExternalTrackerStyle:  "c",
+			},
+			wantFixed: true,
+			wantErr:   false,
+		},
+		{
+			name:      "broken - too many fields",
+			bs:        "&{a b c d}",
+			wantFixed: false,
+			wantErr:   true,
+		},
+		{
+			name:      "broken - wrong format",
+			bs:        "a b c d}",
+			wantFixed: false,
+			wantErr:   true,
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			cfg := &models.ExternalTrackerConfig{}
+			gotFixed, err := fixExternalTrackerConfig16961([]byte(tt.bs), cfg)
+			if (err != nil) != tt.wantErr {
+				t.Errorf("fixExternalTrackerConfig_16961() error = %v, wantErr %v", err, tt.wantErr)
+				return
+			}
+			if gotFixed != tt.wantFixed {
+				t.Errorf("fixExternalTrackerConfig_16961() = %v, want %v", gotFixed, tt.wantFixed)
+			}
+			if cfg.ExternalTrackerFormat != tt.expected.ExternalTrackerFormat {
+				t.Errorf("fixExternalTrackerConfig_16961().ExternalTrackerFormat = %v, want %v", tt.expected.ExternalTrackerFormat, cfg.ExternalTrackerFormat)
+			}
+			if cfg.ExternalTrackerStyle != tt.expected.ExternalTrackerStyle {
+				t.Errorf("fixExternalTrackerConfig_16961().ExternalTrackerStyle = %v, want %v", tt.expected.ExternalTrackerStyle, cfg.ExternalTrackerStyle)
+			}
+			if cfg.ExternalTrackerURL != tt.expected.ExternalTrackerURL {
+				t.Errorf("fixExternalTrackerConfig_16961().ExternalTrackerURL = %v, want %v", tt.expected.ExternalTrackerURL, cfg.ExternalTrackerURL)
+			}
+		})
+	}
+}
+
+func Test_fixPullRequestsConfig_16961(t *testing.T) {
+	tests := []struct {
+		name      string
+		bs        string
+		expected  models.PullRequestsConfig
+		wantFixed bool
+		wantErr   bool
+	}{
+		{
+			name: "normal",
+			bs:   `{"IgnoreWhitespaceConflicts":false,"AllowMerge":false,"AllowRebase":false,"AllowRebaseMerge":false,"AllowSquash":false,"AllowManualMerge":false,"AutodetectManualMerge":false,"DefaultDeleteBranchAfterMerge":false,"DefaultMergeStyle":""}`,
+		},
+		{
+			name: "broken - 1.14",
+			bs:   `&{%!s(bool=false) %!s(bool=true) %!s(bool=true) %!s(bool=true) %!s(bool=true) %!s(bool=false) %!s(bool=false)}`,
+			expected: models.PullRequestsConfig{
+				IgnoreWhitespaceConflicts: false,
+				AllowMerge:                true,
+				AllowRebase:               true,
+				AllowRebaseMerge:          true,
+				AllowSquash:               true,
+				AllowManualMerge:          false,
+				AutodetectManualMerge:     false,
+			},
+			wantFixed: true,
+		},
+		{
+			name: "broken - 1.15",
+			bs:   `&{%!s(bool=false) %!s(bool=true) %!s(bool=true) %!s(bool=true) %!s(bool=true) %!s(bool=false) %!s(bool=false) %!s(bool=false) merge}`,
+			expected: models.PullRequestsConfig{
+				AllowMerge:        true,
+				AllowRebase:       true,
+				AllowRebaseMerge:  true,
+				AllowSquash:       true,
+				DefaultMergeStyle: models.MergeStyleMerge,
+			},
+			wantFixed: true,
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			cfg := &models.PullRequestsConfig{}
+			gotFixed, err := fixPullRequestsConfig16961([]byte(tt.bs), cfg)
+			if (err != nil) != tt.wantErr {
+				t.Errorf("fixPullRequestsConfig_16961() error = %v, wantErr %v", err, tt.wantErr)
+				return
+			}
+			if gotFixed != tt.wantFixed {
+				t.Errorf("fixPullRequestsConfig_16961() = %v, want %v", gotFixed, tt.wantFixed)
+			}
+			assert.EqualValues(t, &tt.expected, cfg)
+		})
+	}
+}
+
+func Test_fixIssuesConfig_16961(t *testing.T) {
+	tests := []struct {
+		name      string
+		bs        string
+		expected  models.IssuesConfig
+		wantFixed bool
+		wantErr   bool
+	}{
+		{
+			name: "normal",
+			bs:   `{"EnableTimetracker":true,"AllowOnlyContributorsToTrackTime":true,"EnableDependencies":true}`,
+			expected: models.IssuesConfig{
+				EnableTimetracker:                true,
+				AllowOnlyContributorsToTrackTime: true,
+				EnableDependencies:               true,
+			},
+		},
+		{
+			name: "broken",
+			bs:   `&{%!s(bool=true) %!s(bool=true) %!s(bool=true)}`,
+			expected: models.IssuesConfig{
+				EnableTimetracker:                true,
+				AllowOnlyContributorsToTrackTime: true,
+				EnableDependencies:               true,
+			},
+			wantFixed: true,
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			cfg := &models.IssuesConfig{}
+			gotFixed, err := fixIssuesConfig16961([]byte(tt.bs), cfg)
+			if (err != nil) != tt.wantErr {
+				t.Errorf("fixIssuesConfig_16961() error = %v, wantErr %v", err, tt.wantErr)
+				return
+			}
+			if gotFixed != tt.wantFixed {
+				t.Errorf("fixIssuesConfig_16961() = %v, want %v", gotFixed, tt.wantFixed)
+			}
+			assert.EqualValues(t, &tt.expected, cfg)
+		})
+	}
+}

modules/doctor/storage.go

@@ -0,0 +1,76 @@
+// Copyright 2021 The Gitea Authors. All rights reserved.
+// Use of this source code is governed by a MIT-style
+// license that can be found in the LICENSE file.
+
+package doctor
+
+import (
+	"code.gitea.io/gitea/models"
+	"code.gitea.io/gitea/modules/log"
+	"code.gitea.io/gitea/modules/storage"
+)
+
+func checkAttachmentStorageFiles(logger log.Logger, autofix bool) error {
+	var total, garbageNum int
+	var deletePaths []string
+	if err := storage.Attachments.IterateObjects(func(p string, obj storage.Object) error {
+		defer obj.Close()
+
+		total++
+		stat, err := obj.Stat()
+		if err != nil {
+			return err
+		}
+		exist, err := models.ExistAttachmentsByUUID(stat.Name())
+		if err != nil {
+			return err
+		}
+		if !exist {
+			garbageNum++
+			if autofix {
+				deletePaths = append(deletePaths, p)
+			}
+		}
+		return nil
+	}); err != nil {
+		logger.Error("storage.Attachments.IterateObjects failed: %v", err)
+		return err
+	}
+
+	if garbageNum > 0 {
+		if autofix {
+			var deletedNum int
+			for _, p := range deletePaths {
+				if err := storage.Attachments.Delete(p); err != nil {
+					log.Error("Delete attachment %s failed: %v", p, err)
+				} else {
+					deletedNum++
+				}
+			}
+			logger.Info("%d missed information attachment detected, %d deleted.", garbageNum, deletedNum)
+		} else {
+			logger.Warn("Checked %d attachment, %d missed information.", total, garbageNum)
+		}
+	}
+	return nil
+}
+
+func checkStorageFiles(logger log.Logger, autofix bool) error {
+	if err := storage.Init(); err != nil {
+		logger.Error("storage.Init failed: %v", err)
+		return err
+	}
+	return checkAttachmentStorageFiles(logger, autofix)
+}
+
+func init() {
+	Register(&Check{
+		Title:                      "Check if there is garbage storage files",
+		Name:                       "storages",
+		IsDefault:                  false,
+		Run:                        checkStorageFiles,
+		AbortIfFailed:              false,
+		SkipDatabaseInitialization: false,
+		Priority:                   1,
+	})
+}

modules/git/batch_reader.go

@@ -8,8 +8,10 @@ import (
 	"bufio"
 	"bytes"
 	"context"
+	"fmt"
 	"io"
 	"math"
+	"runtime"
 	"strconv"
 	"strings"
 
@@ -40,9 +42,14 @@ func CatFileBatchCheck(repoPath string) (WriteCloserError, *bufio.Reader, func()
 		<-closed
 	}
 
+	_, filename, line, _ := runtime.Caller(2)
+	filename = strings.TrimPrefix(filename, callerPrefix)
+
 	go func() {
 		stderr := strings.Builder{}
-		err := NewCommandContext(ctx, "cat-file", "--batch-check").RunInDirFullPipeline(repoPath, batchStdoutWriter, &stderr, batchStdinReader)
+		err := NewCommandContext(ctx, "cat-file", "--batch-check").
+			SetDescription(fmt.Sprintf("%s cat-file --batch-check [repo_path: %s] (%s:%d)", GitExecutable, repoPath, filename, line)).
+			RunInDirFullPipeline(repoPath, batchStdoutWriter, &stderr, batchStdinReader)
 		if err != nil {
 			_ = batchStdoutWriter.CloseWithError(ConcatenateError(err, (&stderr).String()))
 			_ = batchStdinReader.CloseWithError(ConcatenateError(err, (&stderr).String()))
@@ -76,9 +83,14 @@ func CatFileBatch(repoPath string) (WriteCloserError, *bufio.Reader, func()) {
 		<-closed
 	}
 
+	_, filename, line, _ := runtime.Caller(2)
+	filename = strings.TrimPrefix(filename, callerPrefix)
+
 	go func() {
 		stderr := strings.Builder{}
-		err := NewCommandContext(ctx, "cat-file", "--batch").RunInDirFullPipeline(repoPath, batchStdoutWriter, &stderr, batchStdinReader)
+		err := NewCommandContext(ctx, "cat-file", "--batch").
+			SetDescription(fmt.Sprintf("%s cat-file --batch [repo_path: %s] (%s:%d)", GitExecutable, repoPath, filename, line)).
+			RunInDirFullPipeline(repoPath, batchStdoutWriter, &stderr, batchStdinReader)
 		if err != nil {
 			_ = batchStdoutWriter.CloseWithError(ConcatenateError(err, (&stderr).String()))
 			_ = batchStdinReader.CloseWithError(ConcatenateError(err, (&stderr).String()))
@@ -292,3 +304,10 @@ func ParseTreeLine(rd *bufio.Reader, modeBuf, fnameBuf, shaBuf []byte) (mode, fn
 	sha = shaBuf
 	return
 }
+
+var callerPrefix string
+
+func init() {
+	_, filename, _, _ := runtime.Caller(0)
+	callerPrefix = strings.TrimSuffix(filename, "modules/git/batch_reader.go")
+}

modules/git/git.go

@@ -188,6 +188,12 @@ func Init(ctx context.Context) error {
 			return err
 		}
 	}
+	if setting.Git.DisableCoreProtectNTFS {
+		if err := checkAndSetConfig("core.protectntfs", "false", true); err != nil {
+			return err
+		}
+		GlobalCommandArgs = append(GlobalCommandArgs, "-c", "core.protectntfs=false")
+	}
 	return nil
 }
 

modules/git/log_name_status.go

@@ -18,11 +18,16 @@ import (
 )
 
 // LogNameStatusRepo opens git log --raw in the provided repo and returns a stdin pipe, a stdout reader and cancel function
-func LogNameStatusRepo(repository, head, treepath string, paths ...string) (*bufio.Reader, func()) {
+func LogNameStatusRepo(ctx context.Context, repository, head, treepath string, paths ...string) (*bufio.Reader, func()) {
 	// We often want to feed the commits in order into cat-file --batch, followed by their trees and sub trees as necessary.
 	// so let's create a batch stdin and stdout
 	stdoutReader, stdoutWriter := nio.Pipe(buffer.New(32 * 1024))
+
+	// Lets also create a context so that we can absolutely ensure that the command should die when we're done
+	ctx, ctxCancel := context.WithCancel(ctx)
+
 	cancel := func() {
+		ctxCancel()
 		_ = stdoutReader.Close()
 		_ = stdoutWriter.Close()
 	}
@@ -50,7 +55,7 @@ func LogNameStatusRepo(repository, head, treepath string, paths ...string) (*buf
 
 	go func() {
 		stderr := strings.Builder{}
-		err := NewCommand(args...).RunInDirFullPipeline(repository, stdoutWriter, &stderr, nil)
+		err := NewCommandContext(ctx, args...).RunInDirFullPipeline(repository, stdoutWriter, &stderr, nil)
 		if err != nil {
 			_ = stdoutWriter.CloseWithError(ConcatenateError(err, (&stderr).String()))
 		} else {
@@ -75,8 +80,8 @@ type LogNameStatusRepoParser struct {
 }
 
 // NewLogNameStatusRepoParser returns a new parser for a git log raw output
-func NewLogNameStatusRepoParser(repository, head, treepath string, paths ...string) *LogNameStatusRepoParser {
-	rd, cancel := LogNameStatusRepo(repository, head, treepath, paths...)
+func NewLogNameStatusRepoParser(ctx context.Context, repository, head, treepath string, paths ...string) *LogNameStatusRepoParser {
+	rd, cancel := LogNameStatusRepo(ctx, repository, head, treepath, paths...)
 	return &LogNameStatusRepoParser{
 		treepath: treepath,
 		paths:    paths,
@@ -311,8 +316,11 @@ func WalkGitLog(ctx context.Context, repo *Repository, head *Commit, treepath st
 		}
 	}
 
-	g := NewLogNameStatusRepoParser(repo.Path, head.ID.String(), treepath, paths...)
-	defer g.Close()
+	g := NewLogNameStatusRepoParser(ctx, repo.Path, head.ID.String(), treepath, paths...)
+	// don't use defer g.Close() here as g may change its value - instead wrap in a func
+	defer func() {
+		g.Close()
+	}()
 
 	results := make([]string, len(paths))
 	remaining := len(paths)
@@ -331,6 +339,7 @@ heaploop:
 	for {
 		select {
 		case <-ctx.Done():
+			g.Close()
 			return nil, ctx.Err()
 		default:
 		}
@@ -380,7 +389,7 @@ heaploop:
 						remainingPaths = append(remainingPaths, pth)
 					}
 				}
-				g = NewLogNameStatusRepoParser(repo.Path, lastEmptyParent, treepath, remainingPaths...)
+				g = NewLogNameStatusRepoParser(ctx, repo.Path, lastEmptyParent, treepath, remainingPaths...)
 				parentRemaining = map[string]bool{}
 				nextRestart = (remaining * 3) / 4
 				continue heaploop

modules/graceful/server.go

@@ -229,7 +229,7 @@ func (wl *wrappedListener) Accept() (net.Conn, error) {
 
 	closed := int32(0)
 
-	c = wrappedConn{
+	c = &wrappedConn{
 		Conn:                 c,
 		server:               wl.server,
 		closed:               &closed,
@@ -264,7 +264,7 @@ type wrappedConn struct {
 	perWritePerKbTimeout time.Duration
 }
 
-func (w wrappedConn) Write(p []byte) (n int, err error) {
+func (w *wrappedConn) Write(p []byte) (n int, err error) {
 	if w.perWriteTimeout > 0 {
 		minTimeout := time.Duration(len(p)/1024) * w.perWritePerKbTimeout
 		minDeadline := time.Now().Add(minTimeout).Add(w.perWriteTimeout)
@@ -278,7 +278,7 @@ func (w wrappedConn) Write(p []byte) (n int, err error) {
 	return w.Conn.Write(p)
 }
 
-func (w wrappedConn) Close() error {
+func (w *wrappedConn) Close() error {
 	if atomic.CompareAndSwapInt32(w.closed, 0, 1) {
 		defer func() {
 			if err := recover(); err != nil {

modules/highlight/highlight.go

@@ -66,17 +66,6 @@ func Code(fileName, code string) string {
 	if len(code) > sizeLimit {
 		return code
 	}
-	formatter := html.New(html.WithClasses(true),
-		html.WithLineNumbers(false),
-		html.PreventSurroundingPre(true),
-	)
-	if formatter == nil {
-		log.Error("Couldn't create chroma formatter")
-		return code
-	}
-
-	htmlbuf := bytes.Buffer{}
-	htmlw := bufio.NewWriter(&htmlbuf)
 
 	var lexer chroma.Lexer
 	if val, ok := highlightMapping[filepath.Ext(fileName)]; ok {
@@ -97,6 +86,18 @@ func Code(fileName, code string) string {
 		}
 		cache.Add(fileName, lexer)
 	}
+	return CodeFromLexer(lexer, code)
+}
+
+// CodeFromLexer returns a HTML version of code string with chroma syntax highlighting classes
+func CodeFromLexer(lexer chroma.Lexer, code string) string {
+	formatter := html.New(html.WithClasses(true),
+		html.WithLineNumbers(false),
+		html.PreventSurroundingPre(true),
+	)
+
+	htmlbuf := bytes.Buffer{}
+	htmlw := bufio.NewWriter(&htmlbuf)
 
 	iterator, err := lexer.Tokenise(nil, string(code))
 	if err != nil {

modules/markup/html.go

@@ -827,7 +827,7 @@ func issueIndexPatternProcessor(ctx *RenderContext, node *html.Node) {
 		reftext := node.Data[ref.RefLocation.Start:ref.RefLocation.End]
 		if exttrack && !ref.IsPull {
 			ctx.Metas["index"] = ref.Issue
-			link = createLink(com.Expand(ctx.Metas["format"], ctx.Metas), reftext, "ref-issue")
+			link = createLink(com.Expand(ctx.Metas["format"], ctx.Metas), reftext, "ref-issue ref-external-issue")
 		} else {
 			// Path determines the type of link that will be rendered. It's unknown at this point whether
 			// the linked item is actually a PR or an issue. Luckily it's of no real consequence because

modules/markup/html_internal_test.go

@@ -96,12 +96,14 @@ func TestRender_IssueIndexPattern2(t *testing.T) {
 	// numeric: render inputs with valid mentions
 	test := func(s, expectedFmt, marker string, indices ...int) {
 		var path, prefix string
+		isExternal := false
 		if marker == "!" {
 			path = "pulls"
 			prefix = "http://localhost:3000/someUser/someRepo/pulls/"
 		} else {
 			path = "issues"
 			prefix = "https://someurl.com/someUser/someRepo/"
+			isExternal = true
 		}
 
 		links := make([]interface{}, len(indices))
@@ -111,8 +113,13 @@ func TestRender_IssueIndexPattern2(t *testing.T) {
 		expectedNil := fmt.Sprintf(expectedFmt, links...)
 		testRenderIssueIndexPattern(t, s, expectedNil, &RenderContext{Metas: localMetas})
 
+		class := "ref-issue"
+		if isExternal {
+			class += " ref-external-issue"
+		}
+
 		for i, index := range indices {
-			links[i] = numericIssueLink(prefix, "ref-issue", index, marker)
+			links[i] = numericIssueLink(prefix, class, index, marker)
 		}
 		expectedNum := fmt.Sprintf(expectedFmt, links...)
 		testRenderIssueIndexPattern(t, s, expectedNum, &RenderContext{Metas: numericMetas})
@@ -178,7 +185,7 @@ func TestRender_IssueIndexPattern4(t *testing.T) {
 	test := func(s, expectedFmt string, names ...string) {
 		links := make([]interface{}, len(names))
 		for i, name := range names {
-			links[i] = alphanumIssueLink("https://someurl.com/someUser/someRepo/", "ref-issue", name)
+			links[i] = alphanumIssueLink("https://someurl.com/someUser/someRepo/", "ref-issue ref-external-issue", name)
 		}
 		expected := fmt.Sprintf(expectedFmt, links...)
 		testRenderIssueIndexPattern(t, s, expected, &RenderContext{Metas: alphanumericMetas})

modules/markup/orgmode/orgmode.go

@@ -12,6 +12,7 @@ import (
 	"strings"
 
 	"code.gitea.io/gitea/modules/highlight"
+	"code.gitea.io/gitea/modules/log"
 	"code.gitea.io/gitea/modules/markup"
 	"code.gitea.io/gitea/modules/setting"
 	"code.gitea.io/gitea/modules/util"
@@ -51,6 +52,12 @@ func (Renderer) SanitizerRules() []setting.MarkupSanitizerRule {
 func Render(ctx *markup.RenderContext, input io.Reader, output io.Writer) error {
 	htmlWriter := org.NewHTMLWriter()
 	htmlWriter.HighlightCodeBlock = func(source, lang string, inline bool) string {
+		defer func() {
+			if err := recover(); err != nil {
+				log.Error("Panic in HighlightCodeBlock: %v\n%s", err, log.Stack(2))
+				panic(err)
+			}
+		}()
 		var w strings.Builder
 		if _, err := w.WriteString(`<pre>`); err != nil {
 			return ""
@@ -80,7 +87,7 @@ func Render(ctx *markup.RenderContext, input io.Reader, output io.Writer) error
 			}
 			lexer = chroma.Coalesce(lexer)
 
-			if _, err := w.WriteString(highlight.Code(lexer.Config().Filenames[0], source)); err != nil {
+			if _, err := w.WriteString(highlight.CodeFromLexer(lexer, source)); err != nil {
 				return ""
 			}
 		}

modules/markup/orgmode/orgmode_test.go

@@ -57,3 +57,29 @@ func TestRender_Images(t *testing.T) {
 	test("[[file:"+url+"]]",
 		"<p><img src=\""+result+"\" alt=\""+result+"\" title=\""+result+"\" /></p>")
 }
+
+func TestRender_Source(t *testing.T) {
+	setting.AppURL = AppURL
+	setting.AppSubURL = AppSubURL
+
+	test := func(input, expected string) {
+		buffer, err := RenderString(&markup.RenderContext{
+			URLPrefix: setting.AppSubURL,
+		}, input)
+		assert.NoError(t, err)
+		assert.Equal(t, strings.TrimSpace(expected), strings.TrimSpace(buffer))
+	}
+
+	test(`#+begin_src go
+// HelloWorld prints "Hello World"
+func HelloWorld() {
+	fmt.Println("Hello World")
+}
+#+end_src
+`, `<div class="src src-go">
+<pre><code class="chroma language-go"><span class="c1">// HelloWorld prints &#34;Hello World&#34;
+</span><span class="c1"></span><span class="kd">func</span> <span class="nf">HelloWorld</span><span class="p">()</span> <span class="p">{</span>
+	<span class="nx">fmt</span><span class="p">.</span><span class="nf">Println</span><span class="p">(</span><span class="s">&#34;Hello World&#34;</span><span class="p">)</span>
+<span class="p">}</span></code></pre>
+</div>`)
+}

modules/markup/sanitizer.go

@@ -66,7 +66,7 @@ func createDefaultPolicy() *bluemonday.Policy {
 	}
 
 	// Allow classes for anchors
-	policy.AllowAttrs("class").Matching(regexp.MustCompile(`ref-issue`)).OnElements("a")
+	policy.AllowAttrs("class").Matching(regexp.MustCompile(`ref-issue( ref-external-issue)?`)).OnElements("a")
 
 	// Allow classes for task lists
 	policy.AllowAttrs("class").Matching(regexp.MustCompile(`task-list-item`)).OnElements("li")

modules/migrations/github.go

@@ -541,6 +541,9 @@ func (g *GithubDownloaderV3) GetAllComments(page, perPage int) ([]*base.Comment,
 		created     = "created"
 		asc         = "asc"
 	)
+	if perPage > g.maxPerPage {
+		perPage = g.maxPerPage
+	}
 	opt := &github.IssueListCommentsOptions{
 		Sort:      &created,
 		Direction: &asc,
@@ -555,7 +558,9 @@ func (g *GithubDownloaderV3) GetAllComments(page, perPage int) ([]*base.Comment,
 	if err != nil {
 		return nil, false, fmt.Errorf("error while listing repos: %v", err)
 	}
-	log.Trace("Request get comments %d/%d, but in fact get %d", perPage, page, len(comments))
+	var isEnd = resp.NextPage == 0
+
+	log.Trace("Request get comments %d/%d, but in fact get %d, next page is %d", perPage, page, len(comments), resp.NextPage)
 	g.rate = &resp.Rate
 	for _, comment := range comments {
 		var email string
@@ -600,7 +605,7 @@ func (g *GithubDownloaderV3) GetAllComments(page, perPage int) ([]*base.Comment,
 		})
 	}
 
-	return allComments, len(allComments) < perPage, nil
+	return allComments, isEnd, nil
 }
 
 // GetPullRequests returns pull requests according page and perPage

modules/notification/webhook/webhook.go

@@ -51,7 +51,7 @@ func (m *webhookNotifier) NotifyIssueClearLabels(doer *models.User, issue *model
 		err = webhook_services.PrepareWebhooks(issue.Repo, models.HookEventPullRequestLabel, &api.PullRequestPayload{
 			Action:      api.HookIssueLabelCleared,
 			Index:       issue.Index,
-			PullRequest: convert.ToAPIPullRequest(issue.PullRequest),
+			PullRequest: convert.ToAPIPullRequest(issue.PullRequest, nil),
 			Repository:  convert.ToRepo(issue.Repo, mode),
 			Sender:      convert.ToUser(doer, nil),
 		})
@@ -145,7 +145,7 @@ func (m *webhookNotifier) NotifyIssueChangeAssignee(doer *models.User, issue *mo
 		issue.PullRequest.Issue = issue
 		apiPullRequest := &api.PullRequestPayload{
 			Index:       issue.Index,
-			PullRequest: convert.ToAPIPullRequest(issue.PullRequest),
+			PullRequest: convert.ToAPIPullRequest(issue.PullRequest, nil),
 			Repository:  convert.ToRepo(issue.Repo, mode),
 			Sender:      convert.ToUser(doer, nil),
 		}
@@ -197,7 +197,7 @@ func (m *webhookNotifier) NotifyIssueChangeTitle(doer *models.User, issue *model
 					From: oldTitle,
 				},
 			},
-			PullRequest: convert.ToAPIPullRequest(issue.PullRequest),
+			PullRequest: convert.ToAPIPullRequest(issue.PullRequest, nil),
 			Repository:  convert.ToRepo(issue.Repo, mode),
 			Sender:      convert.ToUser(doer, nil),
 		})
@@ -232,7 +232,7 @@ func (m *webhookNotifier) NotifyIssueChangeStatus(doer *models.User, issue *mode
 		// Merge pull request calls issue.changeStatus so we need to handle separately.
 		apiPullRequest := &api.PullRequestPayload{
 			Index:       issue.Index,
-			PullRequest: convert.ToAPIPullRequest(issue.PullRequest),
+			PullRequest: convert.ToAPIPullRequest(issue.PullRequest, nil),
 			Repository:  convert.ToRepo(issue.Repo, mode),
 			Sender:      convert.ToUser(doer, nil),
 		}
@@ -301,7 +301,7 @@ func (m *webhookNotifier) NotifyNewPullRequest(pull *models.PullRequest, mention
 	if err := webhook_services.PrepareWebhooks(pull.Issue.Repo, models.HookEventPullRequest, &api.PullRequestPayload{
 		Action:      api.HookIssueOpened,
 		Index:       pull.Issue.Index,
-		PullRequest: convert.ToAPIPullRequest(pull),
+		PullRequest: convert.ToAPIPullRequest(pull, nil),
 		Repository:  convert.ToRepo(pull.Issue.Repo, mode),
 		Sender:      convert.ToUser(pull.Issue.Poster, nil),
 	}); err != nil {
@@ -322,7 +322,7 @@ func (m *webhookNotifier) NotifyIssueChangeContent(doer *models.User, issue *mod
 					From: oldContent,
 				},
 			},
-			PullRequest: convert.ToAPIPullRequest(issue.PullRequest),
+			PullRequest: convert.ToAPIPullRequest(issue.PullRequest, nil),
 			Repository:  convert.ToRepo(issue.Repo, mode),
 			Sender:      convert.ToUser(doer, nil),
 		})
@@ -500,7 +500,7 @@ func (m *webhookNotifier) NotifyIssueChangeLabels(doer *models.User, issue *mode
 		err = webhook_services.PrepareWebhooks(issue.Repo, models.HookEventPullRequestLabel, &api.PullRequestPayload{
 			Action:      api.HookIssueLabelUpdated,
 			Index:       issue.Index,
-			PullRequest: convert.ToAPIPullRequest(issue.PullRequest),
+			PullRequest: convert.ToAPIPullRequest(issue.PullRequest, nil),
 			Repository:  convert.ToRepo(issue.Repo, models.AccessModeNone),
 			Sender:      convert.ToUser(doer, nil),
 		})
@@ -542,7 +542,7 @@ func (m *webhookNotifier) NotifyIssueChangeMilestone(doer *models.User, issue *m
 		err = webhook_services.PrepareWebhooks(issue.Repo, models.HookEventPullRequestMilestone, &api.PullRequestPayload{
 			Action:      hookAction,
 			Index:       issue.Index,
-			PullRequest: convert.ToAPIPullRequest(issue.PullRequest),
+			PullRequest: convert.ToAPIPullRequest(issue.PullRequest, nil),
 			Repository:  convert.ToRepo(issue.Repo, mode),
 			Sender:      convert.ToUser(doer, nil),
 		})
@@ -609,7 +609,7 @@ func (*webhookNotifier) NotifyMergePullRequest(pr *models.PullRequest, doer *mod
 	// Merge pull request calls issue.changeStatus so we need to handle separately.
 	apiPullRequest := &api.PullRequestPayload{
 		Index:       pr.Issue.Index,
-		PullRequest: convert.ToAPIPullRequest(pr),
+		PullRequest: convert.ToAPIPullRequest(pr, nil),
 		Repository:  convert.ToRepo(pr.Issue.Repo, mode),
 		Sender:      convert.ToUser(doer, nil),
 		Action:      api.HookIssueClosed,
@@ -642,7 +642,7 @@ func (m *webhookNotifier) NotifyPullRequestChangeTargetBranch(doer *models.User,
 				From: oldBranch,
 			},
 		},
-		PullRequest: convert.ToAPIPullRequest(issue.PullRequest),
+		PullRequest: convert.ToAPIPullRequest(issue.PullRequest, nil),
 		Repository:  convert.ToRepo(issue.Repo, mode),
 		Sender:      convert.ToUser(doer, nil),
 	})
@@ -681,7 +681,7 @@ func (m *webhookNotifier) NotifyPullRequestReview(pr *models.PullRequest, review
 	if err := webhook_services.PrepareWebhooks(review.Issue.Repo, reviewHookType, &api.PullRequestPayload{
 		Action:      api.HookIssueReviewed,
 		Index:       review.Issue.Index,
-		PullRequest: convert.ToAPIPullRequest(pr),
+		PullRequest: convert.ToAPIPullRequest(pr, nil),
 		Repository:  convert.ToRepo(review.Issue.Repo, mode),
 		Sender:      convert.ToUser(review.Reviewer, nil),
 		Review: &api.ReviewPayload{
@@ -736,7 +736,7 @@ func (m *webhookNotifier) NotifyPullRequestSynchronized(doer *models.User, pr *m
 	if err := webhook_services.PrepareWebhooks(pr.Issue.Repo, models.HookEventPullRequestSync, &api.PullRequestPayload{
 		Action:      api.HookIssueSynchronized,
 		Index:       pr.Issue.Index,
-		PullRequest: convert.ToAPIPullRequest(pr),
+		PullRequest: convert.ToAPIPullRequest(pr, nil),
 		Repository:  convert.ToRepo(pr.Issue.Repo, models.AccessModeNone),
 		Sender:      convert.ToUser(doer, nil),
 	}); err != nil {

modules/repository/adopt.go

@@ -66,6 +66,9 @@ func AdoptRepository(doer, u *models.User, opts models.CreateRepoOptions) (*mode
 		if err := adoptRepository(ctx, repoPath, doer, repo, opts); err != nil {
 			return fmt.Errorf("createDelegateHooks: %v", err)
 		}
+		if err := repo.CheckDaemonExportOKCtx(ctx); err != nil {
+			return fmt.Errorf("checkDaemonExportOK: %v", err)
+		}
 
 		// Initialize Issue Labels if selected
 		if len(opts.IssueLabels) > 0 {

modules/repository/create.go

@@ -103,6 +103,10 @@ func CreateRepository(doer, u *models.User, opts models.CreateRepoOptions) (*mod
 			}
 		}
 
+		if err := repo.CheckDaemonExportOKCtx(ctx); err != nil {
+			return fmt.Errorf("checkDaemonExportOK: %v", err)
+		}
+
 		if stdout, err := git.NewCommand("update-server-info").
 			SetDescription(fmt.Sprintf("CreateRepository(git update-server-info): %s", repoPath)).
 			RunInDir(repoPath); err != nil {

modules/repository/fork.go

@@ -13,6 +13,7 @@ import (
 	"code.gitea.io/gitea/modules/git"
 	"code.gitea.io/gitea/modules/log"
 	"code.gitea.io/gitea/modules/structs"
+	"code.gitea.io/gitea/modules/util"
 )
 
 // ForkRepository forks a repository
@@ -45,62 +46,87 @@ func ForkRepository(doer, owner *models.User, oldRepo *models.Repository, name,
 
 	oldRepoPath := oldRepo.RepoPath()
 
+	needsRollback := false
+	rollbackFn := func() {
+		if !needsRollback {
+			return
+		}
+
+		repoPath := models.RepoPath(owner.Name, repo.Name)
+
+		if exists, _ := util.IsExist(repoPath); !exists {
+			return
+		}
+
+		// As the transaction will be failed and hence database changes will be destroyed we only need
+		// to delete the related repository on the filesystem
+		if errDelete := util.RemoveAll(repoPath); errDelete != nil {
+			log.Error("Failed to remove fork repo")
+		}
+	}
+
+	needsRollbackInPanic := true
+	defer func() {
+		panicErr := recover()
+		if panicErr == nil {
+			return
+		}
+
+		if needsRollbackInPanic {
+			rollbackFn()
+		}
+		panic(panicErr)
+	}()
+
 	err = models.WithTx(func(ctx models.DBContext) error {
 		if err = models.CreateRepository(ctx, doer, owner, repo, false); err != nil {
 			return err
 		}
 
-		rollbackRemoveFn := func() {
-			if repo.ID == 0 {
-				return
-			}
-			if errDelete := models.DeleteRepository(doer, owner.ID, repo.ID); errDelete != nil {
-				log.Error("Rollback deleteRepository: %v", errDelete)
-			}
-		}
-
 		if err = models.IncrementRepoForkNum(ctx, oldRepo.ID); err != nil {
-			rollbackRemoveFn()
 			return err
 		}
 
 		// copy lfs files failure should not be ignored
-		if err := models.CopyLFS(ctx, repo, oldRepo); err != nil {
-			rollbackRemoveFn()
+		if err = models.CopyLFS(ctx, repo, oldRepo); err != nil {
 			return err
 		}
 
+		needsRollback = true
+
 		repoPath := models.RepoPath(owner.Name, repo.Name)
-		if stdout, err := git.NewCommand(
-			"clone", "--bare", oldRepoPath, repoPath).
+		if stdout, err := git.NewCommand("clone", "--bare", oldRepoPath, repoPath).
 			SetDescription(fmt.Sprintf("ForkRepository(git clone): %s to %s", oldRepo.FullName(), repo.FullName())).
 			RunInDirTimeout(10*time.Minute, ""); err != nil {
 			log.Error("Fork Repository (git clone) Failed for %v (from %v):\nStdout: %s\nError: %v", repo, oldRepo, stdout, err)
-			rollbackRemoveFn()
 			return fmt.Errorf("git clone: %v", err)
 		}
 
+		if err := repo.CheckDaemonExportOKCtx(ctx); err != nil {
+			return fmt.Errorf("checkDaemonExportOK: %v", err)
+		}
+
 		if stdout, err := git.NewCommand("update-server-info").
 			SetDescription(fmt.Sprintf("ForkRepository(git update-server-info): %s", repo.FullName())).
 			RunInDir(repoPath); err != nil {
 			log.Error("Fork Repository (git update-server-info) failed for %v:\nStdout: %s\nError: %v", repo, stdout, err)
-			rollbackRemoveFn()
 			return fmt.Errorf("git update-server-info: %v", err)
 		}
 
 		if err = createDelegateHooks(repoPath); err != nil {
-			rollbackRemoveFn()
 			return fmt.Errorf("createDelegateHooks: %v", err)
 		}
 		return nil
 	})
+	needsRollbackInPanic = false
 	if err != nil {
+		rollbackFn()
 		return nil, err
 	}
 
 	// even if below operations failed, it could be ignored. And they will be retried
 	ctx := models.DefaultDBContext()
-	if err = repo.UpdateSize(ctx); err != nil {
+	if err := repo.UpdateSize(ctx); err != nil {
 		log.Error("Failed to update size for repository: %v", err)
 	}
 	if err := models.CopyLanguageStat(oldRepo, repo); err != nil {
@@ -109,3 +135,34 @@ func ForkRepository(doer, owner *models.User, oldRepo *models.Repository, name,
 
 	return repo, nil
 }
+
+// ConvertForkToNormalRepository convert the provided repo from a forked repo to normal repo
+func ConvertForkToNormalRepository(repo *models.Repository) error {
+	err := models.WithTx(func(ctx models.DBContext) error {
+		repo, err := models.GetRepositoryByIDCtx(ctx, repo.ID)
+		if err != nil {
+			return err
+		}
+
+		if !repo.IsFork {
+			return nil
+		}
+
+		if err := models.DecrementRepoForkNum(ctx, repo.ForkID); err != nil {
+			log.Error("Unable to decrement repo fork num for old root repo %d of repository %-v whilst converting from fork. Error: %v", repo.ForkID, repo, err)
+			return err
+		}
+
+		repo.IsFork = false
+		repo.ForkID = 0
+
+		if err := models.UpdateRepositoryCtx(ctx, repo, false); err != nil {
+			log.Error("Unable to update repository %-v whilst converting from fork. Error: %v", repo, err)
+			return err
+		}
+
+		return nil
+	})
+
+	return err
+}

modules/repository/generate.go

@@ -275,5 +275,16 @@ func GenerateRepository(ctx models.DBContext, doer, owner *models.User, template
 		return generateRepo, err
 	}
 
+	if err = generateRepo.CheckDaemonExportOKCtx(ctx); err != nil {
+		return generateRepo, fmt.Errorf("checkDaemonExportOK: %v", err)
+	}
+
+	if stdout, err := git.NewCommand("update-server-info").
+		SetDescription(fmt.Sprintf("GenerateRepository(git update-server-info): %s", repoPath)).
+		RunInDir(repoPath); err != nil {
+		log.Error("GenerateRepository(git update-server-info) in %v: Stdout: %s\nError: %v", generateRepo, stdout, err)
+		return generateRepo, fmt.Errorf("error in GenerateRepository(git update-server-info): %v", err)
+	}
+
 	return generateRepo, nil
 }

modules/repository/repo.go

@@ -95,6 +95,21 @@ func MigrateRepositoryGitData(ctx context.Context, u *models.User, repo *models.
 		}
 	}
 
+	if repo.OwnerID == u.ID {
+		repo.Owner = u
+	}
+
+	if err = repo.CheckDaemonExportOK(); err != nil {
+		return repo, fmt.Errorf("checkDaemonExportOK: %v", err)
+	}
+
+	if stdout, err := git.NewCommandContext(ctx, "update-server-info").
+		SetDescription(fmt.Sprintf("MigrateRepositoryGitData(git update-server-info): %s", repoPath)).
+		RunInDir(repoPath); err != nil {
+		log.Error("MigrateRepositoryGitData(git update-server-info) in %v: Stdout: %s\nError: %v", repo, stdout, err)
+		return repo, fmt.Errorf("error in MigrateRepositoryGitData(git update-server-info): %v", err)
+	}
+
 	gitRepo, err := git.OpenRepository(repoPath)
 	if err != nil {
 		return repo, fmt.Errorf("OpenRepository: %v", err)

modules/setting/git.go

@@ -26,6 +26,7 @@ var (
 		EnableAutoGitWireProtocol bool
 		PullRequestPushMessage    bool
 		LargeObjectThreshold      int64
+		DisableCoreProtectNTFS    bool
 		Timeout                   struct {
 			Default int
 			Migrate int

modules/ssh/ssh.go

@@ -316,10 +316,66 @@ func Listen(host string, port int, ciphers []string, keyExchanges []string, macs
 		}
 	}
 
+	// Workaround slightly broken behaviour in x/crypto/ssh/handshake.go:458-463
+	//
+	// Fundamentally the issue here is that HostKeyAlgos make the incorrect assumption
+	// that the PublicKey().Type() matches the signature algorithm.
+	//
+	// Therefore we need to add duplicates for the RSA with different signing algorithms.
+	signers := make([]ssh.Signer, 0, len(srv.HostSigners))
+	for _, signer := range srv.HostSigners {
+		if signer.PublicKey().Type() == "ssh-rsa" {
+			signers = append(signers,
+				&wrapSigner{
+					Signer:    signer,
+					algorithm: gossh.SigAlgoRSASHA2512,
+				},
+				&wrapSigner{
+					Signer:    signer,
+					algorithm: gossh.SigAlgoRSASHA2256,
+				},
+			)
+		}
+		signers = append(signers, signer)
+	}
+	srv.HostSigners = signers
+
 	go listen(&srv)
 
 }
 
+// wrapSigner wraps a signer and overrides its public key type with the provided algorithm
+type wrapSigner struct {
+	ssh.Signer
+	algorithm string
+}
+
+// PublicKey returns an associated PublicKey instance.
+func (s *wrapSigner) PublicKey() gossh.PublicKey {
+	return &wrapPublicKey{
+		PublicKey: s.Signer.PublicKey(),
+		algorithm: s.algorithm,
+	}
+}
+
+// Sign returns raw signature for the given data. This method
+// will apply the hash specified for the keytype to the data using
+// the algorithm assigned for this key
+func (s *wrapSigner) Sign(rand io.Reader, data []byte) (*gossh.Signature, error) {
+	return s.Signer.(gossh.AlgorithmSigner).SignWithAlgorithm(rand, data, s.algorithm)
+}
+
+// wrapPublicKey wraps a PublicKey and overrides its type
+type wrapPublicKey struct {
+	gossh.PublicKey
+	algorithm string
+}
+
+// Type returns the algorithm
+func (k *wrapPublicKey) Type() string {
+	return k.algorithm
+}
+
 // GenKeyPair make a pair of public and private keys for SSH access.
 // Public key is encoded in the format for inclusion in an OpenSSH authorized_keys file.
 // Private Key generated is PEM encoded

modules/storage/minio.go

@@ -151,7 +151,7 @@ type minioFileInfo struct {
 }
 
 func (m minioFileInfo) Name() string {
-	return m.ObjectInfo.Key
+	return path.Base(m.ObjectInfo.Key)
 }
 
 func (m minioFileInfo) Size() int64 {
@@ -219,7 +219,7 @@ func (m *MinioStorage) IterateObjects(fn func(path string, obj Object) error) er
 		}
 		if err := func(object *minio.Object, fn func(path string, obj Object) error) error {
 			defer object.Close()
-			return fn(strings.TrimPrefix(m.basePath, mObjInfo.Key), &minioObject{object})
+			return fn(strings.TrimPrefix(mObjInfo.Key, m.basePath), &minioObject{object})
 		}(object, fn); err != nil {
 			return convertMinioErr(err)
 		}

modules/storage/storage.go

@@ -91,6 +91,7 @@ func Copy(dstStorage ObjectStorage, dstPath string, srcStorage ObjectStorage, sr
 // Clean delete all the objects in this storage
 func Clean(storage ObjectStorage) error {
 	return storage.IterateObjects(func(path string, obj Object) error {
+		_ = obj.Close()
 		return storage.Delete(path)
 	})
 }

modules/task/migrate.go

@@ -93,7 +93,6 @@ func runMigrateTask(t *models.Task) (err error) {
 	}
 
 	opts.MigrateToRepoID = t.RepoID
-	var repo *models.Repository
 
 	ctx, cancel := context.WithCancel(graceful.GetManager().ShutdownContext())
 	defer cancel()
@@ -107,7 +106,7 @@ func runMigrateTask(t *models.Task) (err error) {
 		return
 	}
 
-	repo, err = migrations.MigrateRepository(ctx, t.Doer, t.Owner.Name, *opts, func(format string, args ...interface{}) {
+	t.Repo, err = migrations.MigrateRepository(ctx, t.Doer, t.Owner.Name, *opts, func(format string, args ...interface{}) {
 		message := models.TranslatableMessage{
 			Format: format,
 			Args:   args,
@@ -118,7 +117,7 @@ func runMigrateTask(t *models.Task) (err error) {
 		_ = t.UpdateCols("message")
 	})
 	if err == nil {
-		log.Trace("Repository migrated [%d]: %s/%s", repo.ID, t.Owner.Name, repo.Name)
+		log.Trace("Repository migrated [%d]: %s/%s", t.Repo.ID, t.Owner.Name, t.Repo.Name)
 		return
 	}
 

modules/timeutil/timestamp.go

@@ -13,8 +13,24 @@ import (
 // TimeStamp defines a timestamp
 type TimeStamp int64
 
+// mock is NOT concurrency-safe!!
+var mock time.Time
+
+// Set sets the time to a mocked time.Time
+func Set(now time.Time) {
+	mock = now
+}
+
+// Unset will unset the mocked time.Time
+func Unset() {
+	mock = time.Time{}
+}
+
 // TimeStampNow returns now int64
 func TimeStampNow() TimeStamp {
+	if !mock.IsZero() {
+		return TimeStamp(mock.Unix())
+	}
 	return TimeStamp(time.Now().Unix())
 }
 

options/locale/locale_bg-BG.ini

@@ -534,7 +534,7 @@ migrate.clone_address_desc=HTTP(S) или Git URL за клониране на
 migrate.clone_local_path=или път към локален сървър
 migrate.permission_denied=Недостатъчни права за импорт на локални хранилища.
 migrate.failed=Грешка при миграция: %v
-migrated_from_fake=Мигриран от %[1]с
+migrated_from_fake=Мигриран от %[1]s
 migrate.migrating=Мигриране от <b>%s</b>...
 migrate.migrating_failed=Мигрирането от <b>%s</b> беше неуспешно.
 

options/locale/locale_de-DE.ini

@@ -326,7 +326,7 @@ hi_user_x=Hallo <b>%s</b>,
 
 activate_account=Bitte aktiviere dein Konto
 activate_account.title=%s, bitte aktiviere dein Konto
-activate_account.text_1=Hallo <b>%[1]s</b>, danke für deine Registrierung bei %[2]!
+activate_account.text_1=Hallo <b>%[1]s</b>, danke für deine Registrierung bei %[2]s!
 activate_account.text_2=Bitte klicke innerhalb von <b>%s</b> auf folgenden Link, um dein Konto zu aktivieren:
 
 activate_email=Bestätige deine E-Mail-Adresse

options/locale/locale_en-US.ini

@@ -345,8 +345,8 @@ reset_password.text = Please click the following link to recover your account wi
 
 register_success = Registration successful
 
-issue_assigned.pull = @%[1]s assigned you to the pull request %[2]s in repository %[3]s.
-issue_assigned.issue = @%[1]s assigned you to the issue %[2]s in repository %[3]s.
+issue_assigned.pull = @%[1]s assigned you to pull request %[2]s in repository %[3]s.
+issue_assigned.issue = @%[1]s assigned you to issue %[2]s in repository %[3]s.
 
 issue.x_mentioned_you = <b>@%s</b> mentioned you:
 issue.action.force_push = <b>%[1]s</b> force-pushed the <b>%[2]s</b> from %[3]s to %[4]s.

options/locale/locale_es-ES.ini

@@ -326,7 +326,7 @@ hi_user_x=Hola <b>%s</b>,
 
 activate_account=Por favor, active su cuenta
 activate_account.title=%s, por favor activa tu cuenta
-activate_account.text_1=¡Hola <b>%[1]s</b>, gracias por registrarse en %[2]!
+activate_account.text_1=¡Hola <b>%[1]s</b>, gracias por registrarse en %[2]s!
 activate_account.text_2=Por favor, haga clic en el siguiente enlace para activar su cuenta dentro de <b>%s</b>:
 
 activate_email=Verifique su correo electrónico

options/locale/locale_nl-NL.ini

@@ -1048,7 +1048,7 @@ issues.action_assignee=Toegewezene
 issues.action_assignee_no_select=Geen verantwoordelijke
 issues.opened_by=%[1]s geopend door <a href="%[2]s">%[3]s</a>
 issues.closed_by=door <a href="%[2]s">%[3]s</a> gesloten %[1]s
-issues.closed_by_fake=met %[2]gesloten %[1]s
+issues.closed_by_fake=met %[2]s gesloten %[1]s
 issues.previous=Vorige
 issues.next=Volgende
 issues.open_title=Open

options/locale/locale_pt-PT.ini

@@ -2245,7 +2245,7 @@ dashboard.task.cancelled=Tarefa: %[1]s cancelada: %[3]s
 dashboard.task.error=Erro na tarefa: %[1]s: %[3]s
 dashboard.task.finished=Tarefa: %[1]s iniciada por %[2]s foi concluída
 dashboard.task.unknown=Tarefa desconhecida: %[1]s
-dashboard.cron.started=Cron iniciado: %[1]
+dashboard.cron.started=Cron iniciado: %[1]s
 dashboard.cron.process=Cron: %[1]s
 dashboard.cron.cancelled=Cron: %s cancelado: %[3]s
 dashboard.cron.error=Erro no cron: %s: %[3]s
@@ -2698,7 +2698,7 @@ notices.delete_success=As notificações do sistema foram eliminadas.
 
 [action]
 create_repo=criou o repositório <a href="%s">%s</a>
-rename_repo=renomeou o repositório de <code>%[1]s</code> para <a href="%[2]s">%[3]</a>
+rename_repo=renomeou o repositório de <code>%[1]s</code> para <a href="%[2]s">%[3]s</a>
 commit_repo=enviou para <a href="%[1]s/src/branch/%[2]s">%[3]s</a> em <a href="%[1]s">%[4]s</a>
 create_issue=`abriu a questão <a href="%s/issues/%s">%s#%[2]s</a>`
 close_issue=`fechou a questão <a href="%s/issues/%s">%s#%[2]s</a>`
@@ -2724,7 +2724,7 @@ reject_pull_request=`sugeriu modificações para <a href="%s/pulls/%s">%s#%[2]s<
 publish_release=`lançou <a href="%s/releases/tag/%s"> "%[4]s" </a> à <a href="%[1]s">%[3]s</a>`
 review_dismissed=`descartou a revisão de <b>%[4]s</b> para <a href="%[1]s/pulls/%[2]s">%[3]s#%[2]s</a>`
 review_dismissed_reason=Motivo:
-create_branch=criou o ramo <a href="%[1]s/src/branch/%[2]s">%[3]s</a> em <a href="%[1]s">%[4]</a>
+create_branch=criou o ramo <a href="%[1]s/src/branch/%[2]s">%[3]s</a> em <a href="%[1]s">%[4]s</a>
 
 [tool]
 ago=há %s

options/locale/locale_ru-RU.ini

@@ -334,7 +334,7 @@ activate_email.title=%s, пожалуйста, подтвердите ваш а
 activate_email.text=Пожалуйста, перейдите по ссылке, чтобы подтвердить ваш адрес электронной почты в течение <b>%s</b>:
 
 register_notify=Добро пожаловать на Gitea
-register_notify.title=%[1], добро пожаловать в %[2]
+register_notify.title=%[1]s, добро пожаловать в %[2]s
 register_notify.text_1=это письмо с вашим подтверждением регистрации в %s!
 register_notify.text_2=Теперь вы можете войти через логин: %s.
 register_notify.text_3=Если эта учетная запись была создана для вас, пожалуйста, сначала <a href="%s">установите пароль</a>.
@@ -345,7 +345,7 @@ reset_password.text=Пожалуйста, перейдите по ссылке,
 
 register_success=Регистрация прошла успешно
 
-issue_assigned.pull=@%[1] назначил вам запрос на слияние %[2] в репозитории %[3].
+issue_assigned.pull=@%[1]s назначил вам запрос на слияние %[2]s в репозитории %[3]s.
 issue_assigned.issue=@%[1]s назначил вам задачу %[2]s в репозитории %[3]s.
 
 issue.x_mentioned_you=<b>@%s</b> упомянул вас:
@@ -1219,8 +1219,8 @@ issues.reopened_at=`переоткрыл(а) эту проблему <a id="%[1]
 issues.commit_ref_at=`упомянул эту задачу в коммите <a id="%[1]s" href="#%[1]s">%[2]s</a>`
 issues.ref_issue_from=`<a href="%[3]s">ссылка на эту проблему %[4]s</a> <a id="%[1]s" href="#%[1]s">%[2]s</a>`
 issues.ref_pull_from=`<a href="%[3]s">ссылается на этот Pull Request %[4]s</a> <a id="%[1]s" href="#%[1]s">%[2]s</a>`
-issues.ref_closing_from=`<a href="%[3]s">ссылается на Pull Request %[4], который закроет эту задачу</a> <a id="%[1]s" href="#%[1]s">%[2]s</a>`
-issues.ref_reopening_from=`<a href="%[3]s">ссылается на Pull Request %[4], который вновь откроет эту задачу</a> <a id="%[1]s" href="#%[1]s">%[2]s</a>`
+issues.ref_closing_from=`<a href="%[3]s">ссылается на Pull Request %[4]s, который закроет эту задачу</a> <a id="%[1]s" href="#%[1]s">%[2]s</a>`
+issues.ref_reopening_from=`<a href="%[3]s">ссылается на Pull Request %[4]s, который вновь откроет эту задачу</a> <a id="%[1]s" href="#%[1]s">%[2]s</a>`
 issues.ref_closed_from=`<a href="%[3]s">закрыл этот запрос %[4]s</a> <a id="%[1]s" href="#%[1]s">%[2]s</a>`
 issues.ref_reopened_from=`<a href="%[3]s">переоткрыл эту задачу %[4]s</a> <a id="%[1]s" href="#%[1]s">%[2]s</a>`
 issues.ref_from=`из %[1]s`
@@ -1303,7 +1303,7 @@ issues.error_modifying_due_date=Не удалось изменить срок в
 issues.error_removing_due_date=Не удалось убрать срок выполнения.
 issues.push_commit_1=добавил(а) %d коммит %s
 issues.push_commits_n=добавил(а) %d коммитов %s
-issues.force_push_codes=`принудительно залито %[1]s от <a class="ui sha" href="%[3]s"><code>%[2]</code></a> к <a class="ui sha" href="%[5]s"><code>%[4]s</code></a> %[6]s`
+issues.force_push_codes=`принудительно залито %[1]s от <a class="ui sha" href="%[3]s"><code>%[2]s</code></a> к <a class="ui sha" href="%[5]s"><code>%[4]s</code></a> %[6]s`
 issues.due_date_form=гггг-мм-дд
 issues.due_date_form_add=Добавить срок выполнения
 issues.due_date_form_edit=Редактировать

options/locale/locale_tr-TR.ini

@@ -1147,7 +1147,7 @@ issues.reopened_at=`<a id="%[1]s" href="#%[1]s">%[2]s</a> konusunu yeniden açt
 issues.commit_ref_at=`<a id="%[1]s" href="#%[1]s">%[2]s</a> işlemesinde bu konuyu işaret etti`
 issues.ref_issue_from=`<a href="%[3]s">bu konuya referansta bulundu %[4]s</a> <a id="%[1]s" href="#%[1]s">%[2]s</a>`
 issues.ref_pull_from=`<a href="%[3]s">bu değişiklik isteğine referansta bulundu %[4]s</a> <a id="%[1]s" href="#%[1]s">%[2]s</a>`
-issues.ref_closing_from=`<a href="%[3]s">bir değişiklik isteğine referansta bulundu %[4] bu konu kapatılacak </a><a id="%[1]s" href="#%[1]s">%[2]s</a>`
+issues.ref_closing_from=`<a href="%[3]s">bir değişiklik isteğine referansta bulundu %[4]s bu konu kapatılacak </a><a id="%[1]s" href="#%[1]s">%[2]s</a>`
 issues.ref_reopening_from=`<a href="%[3]s">bir değişiklik isteğine referansta bulundu %[4]s bu konu yeniden açılacak</a> <a id="%[1]s" href="#%[1]s">%[2]s</a>`
 issues.ref_closed_from=`<a href="%[3]s">bu konuyu kapat%[4]s</a> <a id="%[1]s" href="#%[1]s">%[2]s</a>`
 issues.ref_reopened_from=`<a href="%[3]s">konuyu yeniden aç%[4]s</a> <a id="%[1]s" href="#%[1]s">%[2]s</a>`

options/locale/locale_uk-UA.ini

@@ -348,7 +348,7 @@ issue.x_mentioned_you=<b>@%s</b> згадав вас:
 issue.action.force_push=<b>%[1]s</b> force-pushed <b>%[2]s</b> з %[3]s в %[4]s.
 issue.action.push_n=<b>@%[1]s</b> відправив %[3]d коміти до %[2]s
 issue.action.close=<b>@%[1]s</b> закрито #%[2]d.
-issue.action.reopen=<b>@%[1]</b> заново відкрив #%[2]d.
+issue.action.reopen=<b>@%[1]s</b> заново відкрив #%[2]d.
 issue.action.merge=<b>@%[1]s</b> об'єднав #%[2]d до %[3]s.
 issue.action.approve=<b>@%[1]s</b> затвердили цей запит на злиття.
 issue.action.reject=<b>@%[1]s</b> запитують зміни на цей запит на злиття.
@@ -555,7 +555,7 @@ delete_email=Видалити
 email_deletion=Видалити адресу електронної пошти
 email_deletion_desc=Електронна адреса та пов'язана з нею інформація буде видалена з вашого облікового запису. Git коміти, здійснені через цю електронну адресу, залишиться без змін. Продовжити?
 email_deletion_success=Адресу електронної пошти було видалено.
-theme_update_success=Тему оновлено. 
+theme_update_success=Тему оновлено.
 theme_update_error=Вибрана тема не існує.
 openid_deletion=Видалити адресу OpenID
 openid_deletion_desc=Видалення цієї OpenID-адреси з вашого облікового запису забороняє вам входити з ним. Продовжити?
@@ -1171,7 +1171,7 @@ issues.action_milestone_no_select=Етап відсутній
 issues.action_assignee=Виконавець
 issues.action_assignee_no_select=Немає виконавеця
 issues.opened_by=%[1]s відкрито <a href="%[2]s">%[3]s</a>
-pulls.merged_by=до <a href="%[2]s">%[3]</a> злито %[1]s
+pulls.merged_by=до <a href="%[2]s">%[3]s</a> злито %[1]s
 pulls.merged_by_fake=%[2]s об'єднаний %[1]s
 issues.closed_by=закрито <a href="%[2]s">%[3]s</a> %[1]s
 issues.opened_by_fake=%[2]s відкрив(ла) %[1]s
@@ -1285,7 +1285,7 @@ issues.error_modifying_due_date=Не вдалося змінити дату за
 issues.error_removing_due_date=Не вдалося видалити дату завершення.
 issues.push_commit_1=додав %d коміт %s
 issues.push_commits_n=додав %d коміти(-ів) %s
-issues.force_push_codes=`примусово залито %[1]s з <a class="ui sha" href="%[3]s"><code>%[2]</code></a> до <a class="ui sha" href="%[5]s"><code>%[4]s</code></a> %[6]s`
+issues.force_push_codes=`примусово залито %[1]s з <a class="ui sha" href="%[3]s"><code>%[2]s</code></a> до <a class="ui sha" href="%[5]s"><code>%[4]s</code></a> %[6]s`
 issues.due_date_form=рррр-мм-дд
 issues.due_date_form_add=Додати дату завершення
 issues.due_date_form_edit=Редагувати
@@ -2221,7 +2221,7 @@ dashboard.clean_unbind_oauth_success=Всі незавершені зв'язки
 dashboard.task.started=Запущено завдання: %[1]s
 dashboard.task.process=Завдання: %[1]s
 dashboard.task.cancelled=Завдання: %[1]s скасовано: %[3]s
-dashboard.task.error=Помилка у завданні: %[1]:%[3]s
+dashboard.task.error=Помилка у завданні: %[1]s :%[3]s
 dashboard.task.finished=Завершилося завдання, яке запустив %[2]s: %[1]s
 dashboard.task.unknown=Невідоме завдання: %[1]s
 dashboard.cron.started=Запущено Cron: %[1]s
@@ -2701,7 +2701,7 @@ mirror_sync_delete=синхронізовано й видалено посила
 approve_pull_request=`схвалив <a href="%s/pulls/%s">%s#%[2]s</a>`
 reject_pull_request=`запропонував зміни до <a href="%s/pulls/%s">%s#%[2]s</a>`
 publish_release=`опублікував випуск <a href="%s/releases/tag/%s"> "%[4]s" </a> з <a href="%[1]s">%[3]s</a>`
-review_dismissed=`відхилений відгук від <b>%[4]</b> у <a href="%[1]s/pulls/%[2]s">%[3]s#%[2]s</a>`
+review_dismissed=`відхилений відгук від <b>%[4]s</b> у <a href="%[1]s/pulls/%[2]s">%[3]s#%[2]s</a>`
 review_dismissed_reason=Причина:
 create_branch=створено гілку <a href="%[1]s/src/branch/%[2]s">%[3]s</a> у <a href="%[1]s">%[4]s</a>
 

options/locale/locale_zh-CN.ini

@@ -1380,7 +1380,7 @@ pulls.filter_branch=过滤分支
 pulls.no_results=未找到结果
 pulls.nothing_to_compare=分支内容相同，无需创建合并请求。
 pulls.nothing_to_compare_and_allow_empty_pr=这些分支是相等的，此合并请求将为空。
-pulls.has_pull_request="在这些分支之间的合并请求已存在: <a href="%[1]s/pulls/%[3]d">%[2]s%#[3]d</a>"
+pulls.has_pull_request="在这些分支之间的合并请求已存在: <a href="%[1]s/pulls/%[3]d">%[2]s#%[3]d</a>"
 pulls.create=创建合并请求
 pulls.title_desc=请求将 %[1]d 次代码提交从 <code>%[2]s</code> 合并至 <code id="branch_target">%[3]s</code>
 pulls.merged_title_desc=于 %[4]s 将 %[1]d 次代码提交从 <code>%[2]s</code>合并至 <code>%[3]s</code>

public/img/svg/gitea-github.svg

@@ -1 +1 @@
-<svg viewBox="0 0 64 64" class="svg gitea-github" width="16" height="16" aria-hidden="true"><linearGradient id="a" x1="30.999" x2="30.999" y1="16" y2="55.342" gradientUnits="userSpaceOnUse" spreadMethod="reflect"><stop offset="0" stop-color="#6dc7ff"/><stop offset="1" stop-color="#e6abff"/></linearGradient><path fill="url(#a)" d="M25.008 56.007c-.003-.368-.006-1.962-.009-3.454l-.003-1.55c-6.729.915-8.358-3.78-8.376-3.83-.934-2.368-2.211-3.045-2.266-3.073l-.124-.072c-.463-.316-1.691-1.157-1.342-2.263.315-.997 1.536-1.1 2.091-1.082 3.074.215 4.63 2.978 4.694 3.095 1.569 2.689 3.964 2.411 5.509 1.844.144-.688.367-1.32.659-1.878-4.956-.879-10.571-3.515-10.571-13.104 0-2.633.82-4.96 2.441-6.929-.362-1.206-.774-3.666.446-6.765l.174-.442.452-.144c.416-.137 2.688-.624 7.359 2.433a24.959 24.959 0 0 1 6.074-.759c2.115.01 4.158.265 6.09.759 4.667-3.058 6.934-2.565 7.351-2.433l.451.145.174.44c1.225 3.098.813 5.559.451 6.766 1.618 1.963 2.438 4.291 2.438 6.929 0 9.591-5.621 12.219-10.588 13.087.563 1.065.868 2.402.868 3.878 0 1.683-.007 7.204-.015 8.402l-2-.014c.008-1.196.015-6.708.015-8.389 0-2.442-.943-3.522-1.35-3.874l-1.73-1.497 2.274-.253c5.205-.578 10.525-2.379 10.525-11.341 0-2.33-.777-4.361-2.31-6.036l-.43-.469.242-.587c.166-.401.894-2.442-.043-5.291-.758.045-2.568.402-5.584 2.447l-.384.259-.445-.123c-1.863-.518-3.938-.796-6.001-.806-2.052.01-4.124.288-5.984.806l-.445.123-.383-.259c-3.019-2.044-4.833-2.404-5.594-2.449-.935 2.851-.206 4.892-.04 5.293l.242.587-.429.469c-1.536 1.681-2.314 3.712-2.314 6.036 0 8.958 5.31 10.77 10.504 11.361l2.252.256-1.708 1.49c-.372.325-1.03 1.112-1.254 2.727l-.075.549-.506.227c-1.321.592-5.839 2.162-8.548-2.485-.015-.025-.544-.945-1.502-1.557.646.639 1.433 1.673 2.068 3.287.066.19 1.357 3.622 7.28 2.339l1.206-.262.012 3.978c.003 1.487.006 3.076.009 3.444l-1.998.014z"/><linearGradient id="b" x1="32" x2="32" y1="5" y2="59.167" gradientUnits="userSpaceOnUse" spreadMethod="reflect"><stop offset="0" stop-color="#1a6dff"/><stop offset="1" stop-color="#c822ff"/></linearGradient><path fill="url(#b)" d="M32 58C17.663 58 6 46.337 6 32S17.663 6 32 6s26 11.663 26 26-11.663 26-26 26zm0-50C18.767 8 8 18.767 8 32s10.767 24 24 24 24-10.767 24-24S45.233 8 32 8z"/></svg>
+<svg viewBox="0 0 64 64" class="svg gitea-github" width="16" height="16" aria-hidden="true"><linearGradient id="gitea-github__a" x1="30.999" x2="30.999" y1="16" y2="55.342" gradientUnits="userSpaceOnUse" spreadMethod="reflect"><stop offset="0" stop-color="#6dc7ff"/><stop offset="1" stop-color="#e6abff"/></linearGradient><path fill="url(#gitea-github__a)" d="M25.008 56.007c-.003-.368-.006-1.962-.009-3.454l-.003-1.55c-6.729.915-8.358-3.78-8.376-3.83-.934-2.368-2.211-3.045-2.266-3.073l-.124-.072c-.463-.316-1.691-1.157-1.342-2.263.315-.997 1.536-1.1 2.091-1.082 3.074.215 4.63 2.978 4.694 3.095 1.569 2.689 3.964 2.411 5.509 1.844.144-.688.367-1.32.659-1.878-4.956-.879-10.571-3.515-10.571-13.104 0-2.633.82-4.96 2.441-6.929-.362-1.206-.774-3.666.446-6.765l.174-.442.452-.144c.416-.137 2.688-.624 7.359 2.433a24.959 24.959 0 0 1 6.074-.759c2.115.01 4.158.265 6.09.759 4.667-3.058 6.934-2.565 7.351-2.433l.451.145.174.44c1.225 3.098.813 5.559.451 6.766 1.618 1.963 2.438 4.291 2.438 6.929 0 9.591-5.621 12.219-10.588 13.087.563 1.065.868 2.402.868 3.878 0 1.683-.007 7.204-.015 8.402l-2-.014c.008-1.196.015-6.708.015-8.389 0-2.442-.943-3.522-1.35-3.874l-1.73-1.497 2.274-.253c5.205-.578 10.525-2.379 10.525-11.341 0-2.33-.777-4.361-2.31-6.036l-.43-.469.242-.587c.166-.401.894-2.442-.043-5.291-.758.045-2.568.402-5.584 2.447l-.384.259-.445-.123c-1.863-.518-3.938-.796-6.001-.806-2.052.01-4.124.288-5.984.806l-.445.123-.383-.259c-3.019-2.044-4.833-2.404-5.594-2.449-.935 2.851-.206 4.892-.04 5.293l.242.587-.429.469c-1.536 1.681-2.314 3.712-2.314 6.036 0 8.958 5.31 10.77 10.504 11.361l2.252.256-1.708 1.49c-.372.325-1.03 1.112-1.254 2.727l-.075.549-.506.227c-1.321.592-5.839 2.162-8.548-2.485-.015-.025-.544-.945-1.502-1.557.646.639 1.433 1.673 2.068 3.287.066.19 1.357 3.622 7.28 2.339l1.206-.262.012 3.978c.003 1.487.006 3.076.009 3.444l-1.998.014z"/><linearGradient id="gitea-github__b" x1="32" x2="32" y1="5" y2="59.167" gradientUnits="userSpaceOnUse" spreadMethod="reflect"><stop offset="0" stop-color="#1a6dff"/><stop offset="1" stop-color="#c822ff"/></linearGradient><path fill="url(#gitea-github__b)" d="M32 58C17.663 58 6 46.337 6 32S17.663 6 32 6s26 11.663 26 26-11.663 26-26 26zm0-50C18.767 8 8 18.767 8 32s10.767 24 24 24 24-10.767 24-24S45.233 8 32 8z"/></svg>

routers/api/v1/repo/file.go

@@ -119,13 +119,15 @@ func GetArchive(ctx *context.APIContext) {
 	//     "$ref": "#/responses/notFound"
 
 	repoPath := models.RepoPath(ctx.Params(":username"), ctx.Params(":reponame"))
-	gitRepo, err := git.OpenRepository(repoPath)
-	if err != nil {
-		ctx.Error(http.StatusInternalServerError, "OpenRepository", err)
-		return
+	if ctx.Repo.GitRepo == nil {
+		gitRepo, err := git.OpenRepository(repoPath)
+		if err != nil {
+			ctx.Error(http.StatusInternalServerError, "OpenRepository", err)
+			return
+		}
+		ctx.Repo.GitRepo = gitRepo
+		defer gitRepo.Close()
 	}
-	ctx.Repo.GitRepo = gitRepo
-	defer gitRepo.Close()
 
 	repo.Download(ctx.Context)
 }

routers/api/v1/repo/issue.go

@@ -752,6 +752,15 @@ func EditIssue(ctx *context.APIContext) {
 		}
 	}
 	if form.State != nil {
+		if issue.IsPull {
+			if pr, err := issue.GetPullRequest(); err != nil {
+				ctx.Error(http.StatusInternalServerError, "GetPullRequest", err)
+				return
+			} else if pr.HasMerged {
+				ctx.Error(http.StatusPreconditionFailed, "MergedPRState", "cannot change state of this pull request, it was already merged")
+				return
+			}
+		}
 		issue.IsClosed = api.StateClosed == api.StateType(*form.State)
 	}
 	statusChangeComment, titleChanged, err := models.UpdateIssueByAPI(issue, ctx.User)

routers/api/v1/repo/pull.go

@@ -115,7 +115,7 @@ func ListPullRequests(ctx *context.APIContext) {
 			ctx.Error(http.StatusInternalServerError, "LoadHeadRepo", err)
 			return
 		}
-		apiPrs[i] = convert.ToAPIPullRequest(prs[i])
+		apiPrs[i] = convert.ToAPIPullRequest(prs[i], ctx.User)
 	}
 
 	ctx.SetLinkHeader(int(maxResults), listOptions.PageSize)
@@ -172,7 +172,7 @@ func GetPullRequest(ctx *context.APIContext) {
 		ctx.Error(http.StatusInternalServerError, "LoadHeadRepo", err)
 		return
 	}
-	ctx.JSON(http.StatusOK, convert.ToAPIPullRequest(pr))
+	ctx.JSON(http.StatusOK, convert.ToAPIPullRequest(pr, ctx.User))
 }
 
 // DownloadPullDiff render a pull's raw diff
@@ -437,7 +437,7 @@ func CreatePullRequest(ctx *context.APIContext) {
 	}
 
 	log.Trace("Pull request created: %d/%d", repo.ID, prIssue.ID)
-	ctx.JSON(http.StatusCreated, convert.ToAPIPullRequest(pr))
+	ctx.JSON(http.StatusCreated, convert.ToAPIPullRequest(pr, ctx.User))
 }
 
 // EditPullRequest does what it says
@@ -584,6 +584,10 @@ func EditPullRequest(ctx *context.APIContext) {
 	}
 
 	if form.State != nil {
+		if pr.HasMerged {
+			ctx.Error(http.StatusPreconditionFailed, "MergedPRState", "cannot change state of this pull request, it was already merged")
+			return
+		}
 		issue.IsClosed = api.StateClosed == api.StateType(*form.State)
 	}
 	statusChangeComment, titleChanged, err := models.UpdateIssueByAPI(issue, ctx.User)
@@ -605,7 +609,7 @@ func EditPullRequest(ctx *context.APIContext) {
 	}
 
 	// change pull target branch
-	if len(form.Base) != 0 && form.Base != pr.BaseBranch {
+	if !pr.HasMerged && len(form.Base) != 0 && form.Base != pr.BaseBranch {
 		if !ctx.Repo.GitRepo.IsBranchExist(form.Base) {
 			ctx.Error(http.StatusNotFound, "NewBaseBranchNotExist", fmt.Errorf("new base '%s' not exist", form.Base))
 			return
@@ -640,7 +644,7 @@ func EditPullRequest(ctx *context.APIContext) {
 	}
 
 	// TODO this should be 200, not 201
-	ctx.JSON(http.StatusCreated, convert.ToAPIPullRequest(pr))
+	ctx.JSON(http.StatusCreated, convert.ToAPIPullRequest(pr, ctx.User))
 }
 
 // IsPullRequestMerged checks if a PR exists given an index
@@ -1130,6 +1134,9 @@ func UpdatePullRequest(ctx *context.APIContext) {
 		if models.IsErrMergeConflicts(err) {
 			ctx.Error(http.StatusConflict, "Update", "merge failed because of conflict")
 			return
+		} else if models.IsErrRebaseConflicts(err) {
+			ctx.Error(http.StatusConflict, "Update", "rebase failed because of conflict")
+			return
 		}
 		ctx.Error(http.StatusInternalServerError, "pull_service.Update", err)
 		return

routers/api/v1/repo/repo.go

@@ -374,16 +374,21 @@ func Generate(ctx *context.APIContext) {
 	ctxUser := ctx.User
 	var err error
 	if form.Owner != ctxUser.Name {
-		ctxUser, err = models.GetOrgByName(form.Owner)
+		ctxUser, err = models.GetUserByName(form.Owner)
 		if err != nil {
-			if models.IsErrOrgNotExist(err) {
+			if models.IsErrUserNotExist(err) {
 				ctx.JSON(http.StatusNotFound, map[string]interface{}{
-					"error": "request owner `" + form.Name + "` is not exist",
+					"error": "request owner `" + form.Owner + "` does not exist",
 				})
 				return
 			}
 
-			ctx.Error(http.StatusInternalServerError, "GetOrgByName", err)
+			ctx.Error(http.StatusInternalServerError, "GetUserByName", err)
+			return
+		}
+
+		if !ctx.User.IsAdmin && !ctxUser.IsOrganization() {
+			ctx.Error(http.StatusForbidden, "", "Only admin can generate repository for other user.")
 			return
 		}
 

routers/private/serv.go

@@ -278,7 +278,12 @@ func ServCommand(ctx *context.PrivateContext) {
 	}
 
 	// Permissions checking:
-	if repoExist && (mode > models.AccessModeRead || repo.IsPrivate || setting.Service.RequireSignInView) {
+	if repoExist &&
+		(mode > models.AccessModeRead ||
+			repo.IsPrivate ||
+			owner.Visibility.IsPrivate() ||
+			user.IsRestricted ||
+			setting.Service.RequireSignInView) {
 		if key.Type == models.KeyTypeDeploy {
 			if deployKey.Mode < mode {
 				ctx.JSON(http.StatusUnauthorized, private.ErrServCommand{

routers/web/base.go

@@ -14,7 +14,6 @@ import (
 	"path/filepath"
 	"strings"
 
-	"code.gitea.io/gitea/models"
 	"code.gitea.io/gitea/modules/context"
 	"code.gitea.io/gitea/modules/httpcache"
 	"code.gitea.io/gitea/modules/log"
@@ -147,15 +146,7 @@ func Recovery() func(next http.Handler) http.Handler {
 						"i18n":       lc,
 					}
 
-					var user *models.User
-					if apiContext := context.GetAPIContext(req); apiContext != nil {
-						user = apiContext.User
-					}
-					if user == nil {
-						if ctx := context.GetContext(req); ctx != nil {
-							user = ctx.User
-						}
-					}
+					var user = context.GetContextUser(req)
 					if user == nil {
 						// Get user from session if logged in - do not attempt to sign-in
 						user = auth.SessionUser(sessionStore)

routers/web/repo/pull.go

@@ -752,6 +752,21 @@ func UpdatePullRequest(ctx *context.Context) {
 			ctx.Flash.Error(flashError)
 			ctx.Redirect(ctx.Repo.RepoLink + "/pulls/" + fmt.Sprint(issue.Index))
 			return
+		} else if models.IsErrRebaseConflicts(err) {
+			conflictError := err.(models.ErrRebaseConflicts)
+			flashError, err := ctx.HTMLString(string(tplAlertDetails), map[string]interface{}{
+				"Message": ctx.Tr("repo.pulls.rebase_conflict", utils.SanitizeFlashErrorString(conflictError.CommitSHA)),
+				"Summary": ctx.Tr("repo.pulls.rebase_conflict_summary"),
+				"Details": utils.SanitizeFlashErrorString(conflictError.StdErr) + "<br>" + utils.SanitizeFlashErrorString(conflictError.StdOut),
+			})
+			if err != nil {
+				ctx.ServerError("UpdatePullRequest.HTMLString", err)
+				return
+			}
+			ctx.Flash.Error(flashError)
+			ctx.Redirect(ctx.Repo.RepoLink + "/pulls/" + fmt.Sprint(issue.Index))
+			return
+
 		}
 		ctx.Flash.Error(err.Error())
 		ctx.Redirect(ctx.Repo.RepoLink + "/pulls/" + fmt.Sprint(issue.Index))
@@ -1292,30 +1307,16 @@ func DownloadPullPatch(ctx *context.Context) {
 
 // DownloadPullDiffOrPatch render a pull's raw diff or patch
 func DownloadPullDiffOrPatch(ctx *context.Context, patch bool) {
-	issue, err := models.GetIssueByIndex(ctx.Repo.Repository.ID, ctx.ParamsInt64(":index"))
+	pr, err := models.GetPullRequestByIndex(ctx.Repo.Repository.ID, ctx.ParamsInt64(":index"))
 	if err != nil {
-		if models.IsErrIssueNotExist(err) {
-			ctx.NotFound("GetIssueByIndex", err)
+		if models.IsErrPullRequestNotExist(err) {
+			ctx.NotFound("GetPullRequestByIndex", err)
 		} else {
-			ctx.ServerError("GetIssueByIndex", err)
+			ctx.ServerError("GetPullRequestByIndex", err)
 		}
 		return
 	}
 
-	// Return not found if it's not a pull request
-	if !issue.IsPull {
-		ctx.NotFound("DownloadPullDiff",
-			fmt.Errorf("Issue is not a pull request"))
-		return
-	}
-
-	if err = issue.LoadPullRequest(); err != nil {
-		ctx.ServerError("LoadPullRequest", err)
-		return
-	}
-
-	pr := issue.PullRequest
-
 	if err := pull_service.DownloadDiffOrPatch(pr, ctx, patch); err != nil {
 		ctx.ServerError("DownloadDiffOrPatch", err)
 		return

routers/web/repo/setting.go

@@ -538,10 +538,8 @@ func SettingsPost(ctx *context.Context) {
 			return
 		}
 
-		repo.IsFork = false
-		repo.ForkID = 0
-		if err := models.UpdateRepository(repo, false); err != nil {
-			log.Error("Unable to update repository %-v whilst converting from fork", repo)
+		if err := repository.ConvertForkToNormalRepository(repo); err != nil {
+			log.Error("Unable to convert repository %-v from fork. Error: %v", repo, err)
 			ctx.ServerError("Convert Fork", err)
 			return
 		}

routers/web/repo/view.go

@@ -353,6 +353,10 @@ func renderDirectory(ctx *context.Context, treeLink string) {
 					}
 				} else {
 					ctx.Data["IsRenderedHTML"] = true
+					buf, err = ioutil.ReadAll(rd)
+					if err != nil {
+						log.Error("ReadAll failed: %v", err)
+					}
 					ctx.Data["FileContent"] = strings.ReplaceAll(
 						gotemplate.HTMLEscapeString(string(buf)), "\n", `<br>`,
 					)

routers/web/user/auth.go

@@ -451,7 +451,7 @@ func U2FSign(ctx *context.Context) {
 	for _, reg := range regs {
 		r, err := reg.Parse()
 		if err != nil {
-			log.Fatal("parsing u2f registration: %v", err)
+			log.Error("parsing u2f registration: %v", err)
 			continue
 		}
 		newCounter, authErr := r.Authenticate(*signResp, *challenge, reg.Counter)
@@ -617,7 +617,7 @@ func SignInOAuthCallback(ctx *context.Context) {
 	}
 
 	if u == nil {
-		if !(setting.Service.DisableRegistration || setting.Service.AllowOnlyInternalRegistration) && setting.OAuth2Client.EnableAutoRegistration {
+		if !setting.Service.AllowOnlyInternalRegistration && setting.OAuth2Client.EnableAutoRegistration {
 			// create new user with details from oauth2 provider
 			var missingFields []string
 			if gothUser.UserID == "" {

routers/web/user/setting/keys.go

@@ -208,7 +208,7 @@ func DeleteKey(ctx *context.Context) {
 			return
 		}
 		if external {
-			ctx.Flash.Error(ctx.Tr("setting.ssh_externally_managed"))
+			ctx.Flash.Error(ctx.Tr("settings.ssh_externally_managed"))
 			ctx.Redirect(setting.AppSubURL + "/user/settings/keys")
 			return
 		}

routers/web/user/setting/security_twofa.go

@@ -32,7 +32,7 @@ func RegenerateScratchTwoFactor(ctx *context.Context) {
 	t, err := models.GetTwoFactorByUID(ctx.User.ID)
 	if err != nil {
 		if models.IsErrTwoFactorNotEnrolled(err) {
-			ctx.Flash.Error(ctx.Tr("setting.twofa_not_enrolled"))
+			ctx.Flash.Error(ctx.Tr("settings.twofa_not_enrolled"))
 			ctx.Redirect(setting.AppSubURL + "/user/settings/security")
 		}
 		ctx.ServerError("SettingsTwoFactor: Failed to GetTwoFactorByUID", err)
@@ -62,7 +62,7 @@ func DisableTwoFactor(ctx *context.Context) {
 	t, err := models.GetTwoFactorByUID(ctx.User.ID)
 	if err != nil {
 		if models.IsErrTwoFactorNotEnrolled(err) {
-			ctx.Flash.Error(ctx.Tr("setting.twofa_not_enrolled"))
+			ctx.Flash.Error(ctx.Tr("settings.twofa_not_enrolled"))
 			ctx.Redirect(setting.AppSubURL + "/user/settings/security")
 		}
 		ctx.ServerError("SettingsTwoFactor: Failed to GetTwoFactorByUID", err)
@@ -150,7 +150,7 @@ func EnrollTwoFactor(ctx *context.Context) {
 	if t != nil {
 		// already enrolled - we should redirect back!
 		log.Warn("Trying to re-enroll %-v in twofa when already enrolled", ctx.User)
-		ctx.Flash.Error(ctx.Tr("setting.twofa_is_enrolled"))
+		ctx.Flash.Error(ctx.Tr("settings.twofa_is_enrolled"))
 		ctx.Redirect(setting.AppSubURL + "/user/settings/security")
 		return
 	}
@@ -175,7 +175,7 @@ func EnrollTwoFactorPost(ctx *context.Context) {
 	t, err := models.GetTwoFactorByUID(ctx.User.ID)
 	if t != nil {
 		// already enrolled
-		ctx.Flash.Error(ctx.Tr("setting.twofa_is_enrolled"))
+		ctx.Flash.Error(ctx.Tr("settings.twofa_is_enrolled"))
 		ctx.Redirect(setting.AppSubURL + "/user/settings/security")
 		return
 	}

services/archiver/archiver.go

@@ -132,9 +132,11 @@ func doArchive(r *ArchiveRequest) (*models.RepoArchiver, error) {
 	if err == nil {
 		if archiver.Status == models.RepoArchiverGenerating {
 			archiver.Status = models.RepoArchiverReady
-			return archiver, models.UpdateRepoArchiverStatus(ctx, archiver)
+			if err = models.UpdateRepoArchiverStatus(ctx, archiver); err != nil {
+				return nil, err
+			}
 		}
-		return archiver, nil
+		return archiver, commiter.Commit()
 	}
 
 	if !errors.Is(err, os.ErrNotExist) {

services/gitdiff/csv.go

@@ -21,10 +21,12 @@ type TableDiffCellType uint8
 
 // TableDiffCellType possible values.
 const (
-	TableDiffCellEqual TableDiffCellType = iota + 1
+	TableDiffCellUnchanged TableDiffCellType = iota + 1
 	TableDiffCellChanged
 	TableDiffCellAdd
 	TableDiffCellDel
+	TableDiffCellMovedUnchanged
+	TableDiffCellMovedChanged
 )
 
 // TableDiffCell represents a cell of a TableDiffRow
@@ -53,6 +55,9 @@ type csvReader struct {
 	eof    bool
 }
 
+// ErrorUndefinedCell is for when a row, column coordinates do not exist in the CSV
+var ErrorUndefinedCell = errors.New("undefined cell")
+
 // createCsvReader creates a csvReader and fills the buffer
 func createCsvReader(reader *csv.Reader, bufferRowCount int) (*csvReader, error) {
 	csv := &csvReader{reader: reader}
@@ -70,7 +75,7 @@ func createCsvReader(reader *csv.Reader, bufferRowCount int) (*csvReader, error)
 
 // GetRow gets a row from the buffer if present or advances the reader to the requested row. On the end of the file only nil gets returned.
 func (csv *csvReader) GetRow(row int) ([]string, error) {
-	if row < len(csv.buffer) {
+	if row < len(csv.buffer) && row >= 0 {
 		return csv.buffer[row], nil
 	}
 	if csv.eof {
@@ -131,7 +136,11 @@ func createCsvDiffSingle(reader *csv.Reader, celltype TableDiffCellType) ([]*Tab
 		}
 		cells := make([]*TableDiffCell, len(row))
 		for j := 0; j < len(row); j++ {
-			cells[j] = &TableDiffCell{LeftCell: row[j], Type: celltype}
+			if celltype == TableDiffCellDel {
+				cells[j] = &TableDiffCell{LeftCell: row[j], Type: celltype}
+			} else {
+				cells[j] = &TableDiffCell{RightCell: row[j], Type: celltype}
+			}
 		}
 		rows = append(rows, &TableDiffRow{RowIdx: i, Cells: cells})
 		i++
@@ -141,185 +150,267 @@ func createCsvDiffSingle(reader *csv.Reader, celltype TableDiffCellType) ([]*Tab
 }
 
 func createCsvDiff(diffFile *DiffFile, baseReader *csv.Reader, headReader *csv.Reader) ([]*TableDiffSection, error) {
-	a, err := createCsvReader(baseReader, maxRowsToInspect)
+	// Given the baseReader and headReader, we are going to create CSV Reader for each, baseCSVReader and b respectively
+	baseCSVReader, err := createCsvReader(baseReader, maxRowsToInspect)
+	if err != nil {
+		return nil, err
+	}
+	headCSVReader, err := createCsvReader(headReader, maxRowsToInspect)
 	if err != nil {
 		return nil, err
 	}
 
-	b, err := createCsvReader(headReader, maxRowsToInspect)
-	if err != nil {
-		return nil, err
+	// Initializing the mappings of base to head (a2bColMap) and head to base (b2aColMap) columns
+	a2bColMap, b2aColMap := getColumnMapping(baseCSVReader, headCSVReader)
+
+	// Determines how many cols there will be in the diff table, which includes deleted columns from base and added columns to base
+	numDiffTableCols := len(a2bColMap) + countUnmappedColumns(b2aColMap)
+	if len(a2bColMap) < len(b2aColMap) {
+		numDiffTableCols = len(b2aColMap) + countUnmappedColumns(a2bColMap)
 	}
 
-	a2b, b2a := getColumnMapping(a, b)
-
-	columns := len(a2b) + countUnmappedColumns(b2a)
-	if len(a2b) < len(b2a) {
-		columns = len(b2a) + countUnmappedColumns(a2b)
-	}
-
-	createDiffRow := func(aline int, bline int) (*TableDiffRow, error) {
-		cells := make([]*TableDiffCell, columns)
-
-		if aline == 0 || bline == 0 {
-			var (
-				row      []string
-				celltype TableDiffCellType
-				err      error
-			)
-			if bline == 0 {
-				row, err = a.GetRow(aline - 1)
-				celltype = TableDiffCellDel
-			} else {
-				row, err = b.GetRow(bline - 1)
-				celltype = TableDiffCellAdd
-			}
+	// createDiffTableRow takes the row # of the `a` line and `b` line of a diff (starting from 1), 0 if the line doesn't exist (undefined)
+	// in the base or head respectively.
+	// Returns a TableDiffRow which has the row index
+	createDiffTableRow := func(aLineNum int, bLineNum int) (*TableDiffRow, error) {
+		// diffTableCells is a row of the diff table. It will have a cells for added, deleted, changed, and unchanged content, thus either
+		// the same size as the head table or bigger
+		diffTableCells := make([]*TableDiffCell, numDiffTableCols)
+		var bRow *[]string
+		if bLineNum > 0 {
+			row, err := headCSVReader.GetRow(bLineNum - 1)
 			if err != nil {
 				return nil, err
 			}
-			if row == nil {
-				return nil, nil
+			bRow = &row
+		}
+		var aRow *[]string
+		if aLineNum > 0 {
+			row, err := baseCSVReader.GetRow(aLineNum - 1)
+			if err != nil {
+				return nil, err
 			}
-			for i := 0; i < len(row); i++ {
-				cells[i] = &TableDiffCell{LeftCell: row[i], Type: celltype}
-			}
-			return &TableDiffRow{RowIdx: bline, Cells: cells}, nil
+			aRow = &row
 		}
-
-		arow, err := a.GetRow(aline - 1)
-		if err != nil {
-			return nil, err
-		}
-		brow, err := b.GetRow(bline - 1)
-		if err != nil {
-			return nil, err
-		}
-		if len(arow) == 0 && len(brow) == 0 {
+		if aRow == nil && bRow == nil {
+			// No content
 			return nil, nil
 		}
 
-		for i := 0; i < len(a2b); i++ {
-			acell, _ := getCell(arow, i)
-			if a2b[i] == unmappedColumn {
-				cells[i] = &TableDiffCell{LeftCell: acell, Type: TableDiffCellDel}
-			} else {
-				bcell, _ := getCell(brow, a2b[i])
+		aIndex := 0      // tracks where we are in the a2bColMap
+		bIndex := 0      // tracks where we are in the b2aColMap
+		colsAdded := 0   // incremented whenever we found a column was added
+		colsDeleted := 0 // incrememted whenever a column was deleted
 
-				celltype := TableDiffCellChanged
-				if acell == bcell {
-					celltype = TableDiffCellEqual
+		// We loop until both the aIndex and bIndex are greater than their col map, which then we are done
+		for aIndex < len(a2bColMap) || bIndex < len(b2aColMap) {
+			// Starting from where aIndex is currently pointing, we see if the map is -1 (dleeted) and if is, create column to note that, increment, and look at the next aIndex
+			for aIndex < len(a2bColMap) && a2bColMap[aIndex] == -1 && (bIndex >= len(b2aColMap) || aIndex <= bIndex) {
+				var aCell string
+				if aRow != nil {
+					if cell, err := getCell(*aRow, aIndex); err != nil {
+						if err != ErrorUndefinedCell {
+							return nil, err
+						}
+					} else {
+						aCell = cell
+					}
+				}
+				diffTableCells[bIndex+colsDeleted] = &TableDiffCell{LeftCell: aCell, Type: TableDiffCellDel}
+				aIndex++
+				colsDeleted++
+			}
+
+			// aIndex is now pointing to a column that also exists in b, or is at the end of a2bColMap. If the former,
+			// we can just increment aIndex until it points to a -1 column or one greater than the current bIndex
+			for aIndex < len(a2bColMap) && a2bColMap[aIndex] != -1 {
+				aIndex++
+			}
+
+			// Starting from where bIndex is currently pointing, we see if the map is -1 (added) and if is, create column to note that, increment, and look at the next aIndex
+			for bIndex < len(b2aColMap) && b2aColMap[bIndex] == -1 && (aIndex >= len(a2bColMap) || bIndex < aIndex) {
+				var bCell string
+				cellType := TableDiffCellAdd
+				if bRow != nil {
+					if cell, err := getCell(*bRow, bIndex); err != nil {
+						if err != ErrorUndefinedCell {
+							return nil, err
+						}
+					} else {
+						bCell = cell
+					}
+				} else {
+					cellType = TableDiffCellDel
+				}
+				diffTableCells[bIndex+colsDeleted] = &TableDiffCell{RightCell: bCell, Type: cellType}
+				bIndex++
+				colsAdded++
+			}
+
+			// aIndex is now pointing to a column that also exists in a, or is at the end of b2aColMap. If the former,
+			// we get the a col and b col values (if they exist), figure out if they are the same or not, and if the column moved, and add it to the diff table
+			for bIndex < len(b2aColMap) && b2aColMap[bIndex] != -1 && (aIndex >= len(a2bColMap) || bIndex < aIndex) {
+				var diffTableCell TableDiffCell
+
+				var aCell *string
+				// get the aCell value if the aRow exists
+				if aRow != nil {
+					if cell, err := getCell(*aRow, b2aColMap[bIndex]); err != nil {
+						if err != ErrorUndefinedCell {
+							return nil, err
+						}
+					} else {
+						aCell = &cell
+						diffTableCell.LeftCell = cell
+					}
+				} else {
+					diffTableCell.Type = TableDiffCellAdd
 				}
 
-				cells[i] = &TableDiffCell{LeftCell: acell, RightCell: bcell, Type: celltype}
-			}
-		}
-		for i := 0; i < len(b2a); i++ {
-			if b2a[i] == unmappedColumn {
-				bcell, _ := getCell(brow, i)
-				cells[i] = &TableDiffCell{LeftCell: bcell, Type: TableDiffCellAdd}
+				var bCell *string
+				// get the bCell value if the bRow exists
+				if bRow != nil {
+					if cell, err := getCell(*bRow, bIndex); err != nil {
+						if err != ErrorUndefinedCell {
+							return nil, err
+						}
+					} else {
+						bCell = &cell
+						diffTableCell.RightCell = cell
+					}
+				} else {
+					diffTableCell.Type = TableDiffCellDel
+				}
+
+				// if both a and b have a row that exists, compare the value and determine if the row has moved
+				if aCell != nil && bCell != nil {
+					moved := ((bIndex + colsDeleted) != (b2aColMap[bIndex] + colsAdded))
+					if *aCell != *bCell {
+						if moved {
+							diffTableCell.Type = TableDiffCellMovedChanged
+						} else {
+							diffTableCell.Type = TableDiffCellChanged
+						}
+					} else {
+						if moved {
+							diffTableCell.Type = TableDiffCellMovedUnchanged
+						} else {
+							diffTableCell.Type = TableDiffCellUnchanged
+						}
+						diffTableCell.LeftCell = ""
+					}
+				}
+
+				// Add the diff column to the diff row
+				diffTableCells[bIndex+colsDeleted] = &diffTableCell
+				bIndex++
 			}
 		}
 
-		return &TableDiffRow{RowIdx: bline, Cells: cells}, nil
+		return &TableDiffRow{RowIdx: bLineNum, Cells: diffTableCells}, nil
 	}
 
-	var sections []*TableDiffSection
+	// diffTableSections are TableDiffSections which represent the diffTableSections we get when doing a diff, each will be its own table in the view
+	var diffTableSections []*TableDiffSection
 
 	for i, section := range diffFile.Sections {
-		var rows []*TableDiffRow
+		// Each section has multiple diffTableRows
+		var diffTableRows []*TableDiffRow
 		lines := tryMergeLines(section.Lines)
+		// Loop through the merged lines to get each row of the CSV diff table for this section
 		for j, line := range lines {
 			if i == 0 && j == 0 && (line[0] != 1 || line[1] != 1) {
-				diffRow, err := createDiffRow(1, 1)
+				diffTableRow, err := createDiffTableRow(1, 1)
 				if err != nil {
 					return nil, err
 				}
-				if diffRow != nil {
-					rows = append(rows, diffRow)
+				if diffTableRow != nil {
+					diffTableRows = append(diffTableRows, diffTableRow)
 				}
 			}
-			diffRow, err := createDiffRow(line[0], line[1])
+			diffTableRow, err := createDiffTableRow(line[0], line[1])
 			if err != nil {
 				return nil, err
 			}
-			if diffRow != nil {
-				rows = append(rows, diffRow)
+			if diffTableRow != nil {
+				diffTableRows = append(diffTableRows, diffTableRow)
 			}
 		}
 
-		if len(rows) > 0 {
-			sections = append(sections, &TableDiffSection{Rows: rows})
+		if len(diffTableRows) > 0 {
+			diffTableSections = append(diffTableSections, &TableDiffSection{Rows: diffTableRows})
 		}
 	}
 
-	return sections, nil
+	return diffTableSections, nil
 }
 
 // getColumnMapping creates a mapping of columns between a and b
-func getColumnMapping(a *csvReader, b *csvReader) ([]int, []int) {
-	arow, _ := a.GetRow(0)
-	brow, _ := b.GetRow(0)
+func getColumnMapping(baseCSVReader *csvReader, headCSVReader *csvReader) ([]int, []int) {
+	baseRow, _ := baseCSVReader.GetRow(0)
+	headRow, _ := headCSVReader.GetRow(0)
 
-	a2b := []int{}
-	b2a := []int{}
+	base2HeadColMap := []int{}
+	head2BaseColMap := []int{}
 
-	if arow != nil {
-		a2b = make([]int, len(arow))
+	if baseRow != nil {
+		base2HeadColMap = make([]int, len(baseRow))
 	}
-	if brow != nil {
-		b2a = make([]int, len(brow))
+	if headRow != nil {
+		head2BaseColMap = make([]int, len(headRow))
 	}
 
-	for i := 0; i < len(b2a); i++ {
-		b2a[i] = unmappedColumn
+	// Initializes all head2base mappings to be unmappedColumn (-1)
+	for i := 0; i < len(head2BaseColMap); i++ {
+		head2BaseColMap[i] = unmappedColumn
 	}
 
-	bcol := 0
-	for i := 0; i < len(a2b); i++ {
-		a2b[i] = unmappedColumn
-
-		acell, ea := getCell(arow, i)
-		if ea == nil {
-			for j := bcol; j < len(b2a); j++ {
-				bcell, eb := getCell(brow, j)
-				if eb == nil && acell == bcell {
-					a2b[i] = j
-					b2a[j] = i
-					bcol = j + 1
-					break
+	// Loops through the baseRow and see if there is a match in the head row
+	for i := 0; i < len(baseRow); i++ {
+		base2HeadColMap[i] = unmappedColumn
+		baseCell, err := getCell(baseRow, i)
+		if err == nil {
+			for j := 0; j < len(headRow); j++ {
+				if head2BaseColMap[j] == -1 {
+					headCell, err := getCell(headRow, j)
+					if err == nil && baseCell == headCell {
+						base2HeadColMap[i] = j
+						head2BaseColMap[j] = i
+						break
+					}
 				}
 			}
 		}
 	}
 
-	tryMapColumnsByContent(a, a2b, b, b2a)
-	tryMapColumnsByContent(b, b2a, a, a2b)
+	tryMapColumnsByContent(baseCSVReader, base2HeadColMap, headCSVReader, head2BaseColMap)
+	tryMapColumnsByContent(headCSVReader, head2BaseColMap, baseCSVReader, base2HeadColMap)
 
-	return a2b, b2a
+	return base2HeadColMap, head2BaseColMap
 }
 
 // tryMapColumnsByContent tries to map missing columns by the content of the first lines.
-func tryMapColumnsByContent(a *csvReader, a2b []int, b *csvReader, b2a []int) {
-	start := 0
-	for i := 0; i < len(a2b); i++ {
-		if a2b[i] == unmappedColumn {
-			if b2a[start] == unmappedColumn {
-				rows := util.Min(maxRowsToInspect, util.Max(0, util.Min(len(a.buffer), len(b.buffer))-1))
+func tryMapColumnsByContent(baseCSVReader *csvReader, base2HeadColMap []int, headCSVReader *csvReader, head2BaseColMap []int) {
+	for i := 0; i < len(base2HeadColMap); i++ {
+		headStart := 0
+		for base2HeadColMap[i] == unmappedColumn && headStart < len(head2BaseColMap) {
+			if head2BaseColMap[headStart] == unmappedColumn {
+				rows := util.Min(maxRowsToInspect, util.Max(0, util.Min(len(baseCSVReader.buffer), len(headCSVReader.buffer))-1))
 				same := 0
 				for j := 1; j <= rows; j++ {
-					acell, ea := getCell(a.buffer[j], i)
-					bcell, eb := getCell(b.buffer[j], start+1)
-					if ea == nil && eb == nil && acell == bcell {
+					baseCell, baseErr := getCell(baseCSVReader.buffer[j], i)
+					headCell, headErr := getCell(headCSVReader.buffer[j], headStart)
+					if baseErr == nil && headErr == nil && baseCell == headCell {
 						same++
 					}
 				}
 				if (float32(same) / float32(rows)) > minRatioToMatch {
-					a2b[i] = start + 1
-					b2a[start+1] = i
+					base2HeadColMap[i] = headStart
+					head2BaseColMap[headStart] = i
 				}
 			}
+			headStart++
 		}
-		start = a2b[i]
 	}
 }
 
@@ -328,7 +419,7 @@ func getCell(row []string, column int) (string, error) {
 	if column < len(row) {
 		return row[column], nil
 	}
-	return "", errors.New("Undefined column")
+	return "", ErrorUndefinedCell
 }
 
 // countUnmappedColumns returns the count of unmapped columns.

services/gitdiff/csv_test.go

@@ -19,9 +19,9 @@ func TestCSVDiff(t *testing.T) {
 		diff  string
 		base  string
 		head  string
-		cells [][2]TableDiffCellType
+		cells [][]TableDiffCellType
 	}{
-		// case 0
+		// case 0 - initial commit of a csv
 		{
 			diff: `diff --git a/unittest.csv b/unittest.csv
 --- a/unittest.csv
@@ -29,11 +29,14 @@ func TestCSVDiff(t *testing.T) {
 @@ -0,0 +1,2 @@
 +col1,col2
 +a,a`,
-			base:  "",
-			head:  "col1,col2\na,a",
-			cells: [][2]TableDiffCellType{{TableDiffCellAdd, TableDiffCellAdd}, {TableDiffCellAdd, TableDiffCellAdd}},
+			base: "",
+			head: `col1,col2
+a,a`,
+			cells: [][]TableDiffCellType{
+				{TableDiffCellAdd, TableDiffCellAdd},
+				{TableDiffCellAdd, TableDiffCellAdd}},
 		},
-		// case 1
+		// case 1 - adding 1 row at end
 		{
 			diff: `diff --git a/unittest.csv b/unittest.csv
 --- a/unittest.csv
@@ -43,11 +46,17 @@ func TestCSVDiff(t *testing.T) {
 -a,a
 +a,a
 +b,b`,
-			base:  "col1,col2\na,a",
-			head:  "col1,col2\na,a\nb,b",
-			cells: [][2]TableDiffCellType{{TableDiffCellEqual, TableDiffCellEqual}, {TableDiffCellEqual, TableDiffCellEqual}, {TableDiffCellAdd, TableDiffCellAdd}},
+			base: `col1,col2
+a,a`,
+			head: `col1,col2
+a,a
+b,b`,
+			cells: [][]TableDiffCellType{
+				{TableDiffCellUnchanged, TableDiffCellUnchanged}, {TableDiffCellUnchanged, TableDiffCellUnchanged},
+				{TableDiffCellAdd, TableDiffCellAdd},
+			},
 		},
-		// case 2
+		// case 2 - row deleted
 		{
 			diff: `diff --git a/unittest.csv b/unittest.csv
 --- a/unittest.csv
@@ -56,11 +65,17 @@ func TestCSVDiff(t *testing.T) {
  col1,col2
 -a,a
  b,b`,
-			base:  "col1,col2\na,a\nb,b",
-			head:  "col1,col2\nb,b",
-			cells: [][2]TableDiffCellType{{TableDiffCellEqual, TableDiffCellEqual}, {TableDiffCellDel, TableDiffCellDel}, {TableDiffCellEqual, TableDiffCellEqual}},
+			base: `col1,col2
+a,a
+b,b`,
+			head: `col1,col2
+b,b`,
+			cells: [][]TableDiffCellType{
+				{TableDiffCellUnchanged, TableDiffCellUnchanged}, {TableDiffCellDel, TableDiffCellDel},
+				{TableDiffCellUnchanged, TableDiffCellUnchanged},
+			},
 		},
-		// case 3
+		// case 3 - row changed
 		{
 			diff: `diff --git a/unittest.csv b/unittest.csv
 --- a/unittest.csv
@@ -69,11 +84,16 @@ func TestCSVDiff(t *testing.T) {
  col1,col2
 -b,b
 +b,c`,
-			base:  "col1,col2\nb,b",
-			head:  "col1,col2\nb,c",
-			cells: [][2]TableDiffCellType{{TableDiffCellEqual, TableDiffCellEqual}, {TableDiffCellEqual, TableDiffCellChanged}},
+			base: `col1,col2
+b,b`,
+			head: `col1,col2
+b,c`,
+			cells: [][]TableDiffCellType{
+				{TableDiffCellUnchanged, TableDiffCellUnchanged},
+				{TableDiffCellUnchanged, TableDiffCellChanged},
+			},
 		},
-		// case 4
+		// case 4 - all deleted
 		{
 			diff: `diff --git a/unittest.csv b/unittest.csv
 --- a/unittest.csv
@@ -81,9 +101,88 @@ func TestCSVDiff(t *testing.T) {
 @@ -1,2 +0,0 @@
 -col1,col2
 -b,c`,
-			base:  "col1,col2\nb,c",
-			head:  "",
-			cells: [][2]TableDiffCellType{{TableDiffCellDel, TableDiffCellDel}, {TableDiffCellDel, TableDiffCellDel}},
+			base: `col1,col2
+b,c`,
+			head: "",
+			cells: [][]TableDiffCellType{
+				{TableDiffCellDel, TableDiffCellDel},
+				{TableDiffCellDel, TableDiffCellDel},
+			},
+		},
+		// case 5 - renames first column
+		{
+			diff: `diff --git a/unittest.csv b/unittest.csv
+--- a/unittest.csv
++++ b/unittest.csv
+@@ -1,3 +1,3 @@
+-col1,col2,col3
++cola,col2,col3
+ a,b,c`,
+			base: `col1,col2,col3
+a,b,c`,
+			head: `cola,col2,col3
+a,b,c`,
+			cells: [][]TableDiffCellType{
+				{TableDiffCellDel, TableDiffCellAdd, TableDiffCellUnchanged, TableDiffCellUnchanged},
+				{TableDiffCellDel, TableDiffCellAdd, TableDiffCellUnchanged, TableDiffCellUnchanged},
+			},
+		},
+		// case 6 - inserts a column after first, deletes last column
+		{
+			diff: `diff --git a/unittest.csv b/unittest.csv
+--- a/unittest.csv
++++ b/unittest.csv
+@@ -1,2 +1,2 @@
+-col1,col2,col3
+-a,b,c
++col1,col1a,col2
++a,d,b`,
+			base: `col1,col2,col3
+a,b,c`,
+			head: `col1,col1a,col2
+a,d,b`,
+			cells: [][]TableDiffCellType{
+				{TableDiffCellUnchanged, TableDiffCellAdd, TableDiffCellDel, TableDiffCellMovedUnchanged},
+				{TableDiffCellUnchanged, TableDiffCellAdd, TableDiffCellDel, TableDiffCellMovedUnchanged},
+			},
+		},
+		// case 7 - deletes first column, inserts column after last
+		{
+			diff: `diff --git a/unittest.csv b/unittest.csv
+--- a/unittest.csv
++++ b/unittest.csv
+@@ -1,2 +1,2 @@
+-col1,col2,col3
+-a,b,c
++col2,col3,col4
++b,c,d`,
+			base: `col1,col2,col3
+a,b,c`,
+			head: `col2,col3,col4
+b,c,d`,
+			cells: [][]TableDiffCellType{
+				{TableDiffCellDel, TableDiffCellUnchanged, TableDiffCellUnchanged, TableDiffCellAdd},
+				{TableDiffCellDel, TableDiffCellUnchanged, TableDiffCellUnchanged, TableDiffCellAdd},
+			},
+		},
+		// case 8 - two columns deleted, 2 added
+		{
+			diff: `diff --git a/unittest.csv b/unittest.csv
+--- a/unittest.csv
++++ b/unittest.csv
+@@ -1,2 +1,2 @@
+-col1,col2,col
+-a,b,c
++col3,col4,col5
++c,d,e`,
+			base: `col1,col2,col3
+a,b,c`,
+			head: `col3,col4,col5
+c,d,e`,
+			cells: [][]TableDiffCellType{
+				{TableDiffCellDel, TableDiffCellMovedUnchanged, TableDiffCellDel, TableDiffCellAdd, TableDiffCellAdd},
+				{TableDiffCellDel, TableDiffCellMovedUnchanged, TableDiffCellDel, TableDiffCellAdd, TableDiffCellAdd},
+			},
 		},
 	}
 
@@ -116,7 +215,7 @@ func TestCSVDiff(t *testing.T) {
 		assert.Len(t, section.Rows, len(c.cells), "case %d: should be %d rows", n, len(c.cells))
 
 		for i, row := range section.Rows {
-			assert.Len(t, row.Cells, 2, "case %d: row %d should have two cells", n, i)
+			assert.Len(t, row.Cells, len(c.cells[i]), "case %d: row %d should have %d cells", n, i, len(c.cells[i]))
 			for j, cell := range row.Cells {
 				assert.Equal(t, c.cells[i][j], cell.Type, "case %d: row %d cell %d should be equal", n, i, j)
 			}

services/gitdiff/gitdiff.go

@@ -996,6 +996,7 @@ func parseHunks(curFile *DiffFile, maxLines, maxLineCharacters int, input *bufio
 
 			// Create a new section to represent this hunk
 			curSection = &DiffSection{}
+			lastLeftIdx = -1
 			curFile.Sections = append(curFile.Sections, curSection)
 
 			lineSectionInfo := getDiffLineSectionInfo(curFile.Name, line, leftLine-1, rightLine-1)
@@ -1036,6 +1037,7 @@ func parseHunks(curFile *DiffFile, maxLines, maxLineCharacters int, input *bufio
 				// Create a new section to represent this hunk
 				curSection = &DiffSection{}
 				curFile.Sections = append(curFile.Sections, curSection)
+				lastLeftIdx = -1
 			}
 			if lastLeftIdx > -1 {
 				diffLine.Match = lastLeftIdx
@@ -1061,6 +1063,7 @@ func parseHunks(curFile *DiffFile, maxLines, maxLineCharacters int, input *bufio
 				// Create a new section to represent this hunk
 				curSection = &DiffSection{}
 				curFile.Sections = append(curFile.Sections, curSection)
+				lastLeftIdx = -1
 			}
 			if len(curSection.Lines) == 0 || curSection.Lines[len(curSection.Lines)-1].Type != DiffLineDel {
 				lastLeftIdx = len(curSection.Lines)
@@ -1121,6 +1124,7 @@ func parseHunks(curFile *DiffFile, maxLines, maxLineCharacters int, input *bufio
 					curFile.IsBin = true
 					curFile.IsLFSFile = true
 					curSection.Lines = nil
+					lastLeftIdx = -1
 				}
 			}
 		}

services/mailer/mail.go

@@ -344,6 +344,16 @@ func sanitizeSubject(subject string) string {
 
 // SendIssueAssignedMail composes and sends issue assigned email
 func SendIssueAssignedMail(issue *models.Issue, doer *models.User, content string, comment *models.Comment, recipients []*models.User) error {
+	if setting.MailService == nil {
+		// No mail service configured
+		return nil
+	}
+
+	if err := issue.LoadRepo(); err != nil {
+		log.Error("Unable to load repo [%d] for issue #%d [%d]. Error: %v", issue.RepoID, issue.Index, issue.ID, err)
+		return err
+	}
+
 	langMap := make(map[string][]*models.User)
 	for _, user := range recipients {
 		langMap[user.Language] = append(langMap[user.Language], user)

templates/mail/issue/assigned.tmpl

@@ -8,7 +8,7 @@
 	<title>{{.Subject}}</title>
 </head>
 
-{{$repo_url := printf "<a href='%s'>%s</a>" .Release.Repo.HTMLURL .Release.Repo.FullName}}
+{{$repo_url := printf "<a href='%s'>%s</a>" .Issue.Repo.HTMLURL .Issue.Repo.FullName}}
 {{$link := printf "<a href='%s'>#%d</a>" .Link .Issue.Index}}
 <body>
 	<p>

templates/mail/issue/default.tmpl

@@ -62,7 +62,7 @@
 		{{end -}}
 		{{- range .ReviewComments}}
 			<hr>
-			{{.i18n.Tr "mail.issue.in_tree_path" .TreePath}}
+			{{$.i18n.Tr "mail.issue.in_tree_path" .TreePath}}
 			<div class="review">
 				<pre>{{.Patch}}</pre>
 				<div>{{.RenderedContent | Safe}}</div>

templates/repo/diff/csv_diff.tmpl

@@ -12,12 +12,18 @@
 						{{if and (eq $i 0) (eq $j 0)}}
 							<th class="line-num">{{.RowIdx}}</th>
 							{{range $j, $cell := $row.Cells}}
-								{{if eq $cell.Type 2}}
+								{{if not $cell}}
+									<th></th>
+								{{else if eq $cell.Type 2}}
 									<th class="modified"><span class="removed-code">{{.LeftCell}}</span> <span class="added-code">{{.RightCell}}</span></th>
 								{{else if eq $cell.Type 3}}
-									<th class="added"><span class="added-code">{{.LeftCell}}</span></th>
+									<th class="added"><span class="added-code">{{.RightCell}}</span></th>
 								{{else if eq $cell.Type 4}}
 									<th class="removed"><span class="removed-code">{{.LeftCell}}</span></th>
+								{{else if eq $cell.Type 5}}
+									<th class="moved">{{.RightCell}}</th>
+								{{else if eq $cell.Type 6}}
+									<th class="moved"><span class="removed-code">{{.LeftCell}}</span> <span class="added-code">{{.RightCell}}</span></th>
 								{{else}}
 									<th>{{.RightCell}}</th>
 								{{end}}
@@ -25,12 +31,18 @@
 						{{else}}
 							<td class="line-num">{{if .RowIdx}}{{.RowIdx}}{{end}}</td>
 							{{range $j, $cell := $row.Cells}}
-								{{if eq $cell.Type 2}}
+								{{if not $cell}}
+									<td></td>
+								{{else if eq $cell.Type 2}}
 									<td class="modified"><span class="removed-code">{{.LeftCell}}</span> <span class="added-code">{{.RightCell}}</span></td>
 								{{else if eq $cell.Type 3}}
-									<td class="added"><span class="added-code">{{.LeftCell}}</span></td>
+									<td class="added"><span class="added-code">{{.RightCell}}</span></td>
 								{{else if eq $cell.Type 4}}
 									<td class="removed"><span class="removed-code">{{.LeftCell}}</span></td>
+								{{else if eq $cell.Type 5}}
+									<td class="moved">{{.RightCell}}</td>
+								{{else if eq $cell.Type 6}}
+									<td class="moved"><span class="removed-code">{{.LeftCell}}</span> <span class="added-code">{{.RightCell}}</span></td>
 								{{else}}
 									<td>{{.RightCell}}</td>
 								{{end}}

templates/repo/diff/image_diff.tmpl

@@ -6,15 +6,15 @@
 		<div class="image-diff" data-path-before="{{$imagePathOld}}" data-path-after="{{$imagePathNew}}">
 			<div class="ui secondary pointing tabular top attached borderless menu stackable new-menu">
 				<div class="new-menu-inner">
-					<a class="item active" data-tab="diff-side-by-side">{{.root.i18n.Tr "repo.diff.image.side_by_side"}}</a>
+					<a class="item active" data-tab="diff-side-by-side-{{ .file.Index }}">{{.root.i18n.Tr "repo.diff.image.side_by_side"}}</a>
 					{{if and .blobBase .blobHead}}
-					<a class="item" data-tab="diff-swipe">{{.root.i18n.Tr "repo.diff.image.swipe"}}</a>
-					<a class="item" data-tab="diff-overlay">{{.root.i18n.Tr "repo.diff.image.overlay"}}</a>
+					<a class="item" data-tab="diff-swipe-{{ .file.Index }}">{{.root.i18n.Tr "repo.diff.image.swipe"}}</a>
+					<a class="item" data-tab="diff-overlay-{{ .file.Index }}">{{.root.i18n.Tr "repo.diff.image.overlay"}}</a>
 					{{end}}
 				</div>
 			</div>
 			<div class="hide">
-				<div class="ui bottom attached tab image-diff-container active" data-tab="diff-side-by-side">
+				<div class="ui bottom attached tab image-diff-container active" data-tab="diff-side-by-side-{{ .file.Index }}">
 					<div class="diff-side-by-side">
 						{{if .blobBase }}
 						<span class="side">
@@ -49,7 +49,7 @@
 					</div>
 				</div>
 				{{if and .blobBase .blobHead}}
-				<div class="ui bottom attached tab image-diff-container" data-tab="diff-swipe">
+				<div class="ui bottom attached tab image-diff-container" data-tab="diff-swipe-{{ .file.Index }}">
 					<div class="diff-swipe">
 						<div class="swipe-frame">
 							<span class="before-container"><img class="image-before" /></span>
@@ -63,7 +63,7 @@
 						</div>
 					</div>
 				</div>
-				<div class="ui bottom attached tab image-diff-container" data-tab="diff-overlay">
+				<div class="ui bottom attached tab image-diff-container" data-tab="diff-overlay-{{ .file.Index }}">
 					<div class="diff-overlay">
 						<div class="overlay-frame">
 							<div class="ui centered">

templates/user/auth/signin.tmpl

@@ -2,7 +2,7 @@
 <div class="page-content user signin{{if .LinkAccountMode}} icon{{end}}">
 	{{template "user/auth/signin_navbar" .}}
 	<div class="ui middle very relaxed page grid">
-		<div class="ui container column">
+		<div class="ui container column fluid">
 			{{template "user/auth/signin_inner" .}}
 		</div>
 	</div>

templates/user/auth/signup_inner.tmpl

@@ -1,4 +1,4 @@
-<div class="ui container column{{if .LinkAccountMode}} icon{{end}}">
+<div class="ui container column fluid{{if .LinkAccountMode}} icon{{end}}">
 	<h4 class="ui top attached header center">
 		{{if .LinkAccountMode}}
 			{{.i18n.Tr "auth.oauth_signup_title"}}

vendor/github.com/microcosm-cc/bluemonday/.editorconfig

@@ -0,0 +1,4 @@
+root = true
+
+[*]
+end_of_line = lf

vendor/github.com/microcosm-cc/bluemonday/.gitattributes

@@ -0,0 +1 @@
+* text=auto eol=lf

vendor/github.com/microcosm-cc/bluemonday/CREDITS.md

@@ -4,4 +4,5 @@
 1. Andrew Krasichkov @buglloc https://github.com/buglloc
 1. Mike Samuel mikesamuel@gmail.com
 1. Dmitri Shuralyov shurcooL@gmail.com
-1. https://github.com/opennota
+1. opennota https://github.com/opennota https://gitlab.com/opennota
+1. Tom Anthony https://www.tomanthony.co.uk/

vendor/github.com/microcosm-cc/bluemonday/Makefile

@@ -3,6 +3,7 @@
 #   all:          Builds the code locally after testing
 #
 #   fmt:          Formats the source files
+#   fmt-check:    Check if the source files are formated
 #   build:        Builds the code locally
 #   vet:          Vets the code
 #   lint:         Runs lint over the code (you do not need to fix everything)
@@ -11,6 +12,8 @@
 #
 #   install:      Builds, tests and installs the code locally
 
+GOFILES_NOVENDOR = $(shell find . -type f -name '*.go' -not -path "./vendor/*" -not -path "./.git/*")
+
 .PHONY: all fmt build vet lint test cover install
 
 # The first target is always the default action if `make` is called without
@@ -19,7 +22,10 @@
 all: fmt vet test install
 
 fmt:
-	@gofmt -s -w ./$*
+	@gofmt -s -w ${GOFILES_NOVENDOR}
+
+fmt-check:
+	@([ -z "$(shell gofmt -d $(GOFILES_NOVENDOR) | head)" ]) || (echo "Source is unformatted"; exit 1)
 
 build:
 	@go build

vendor/github.com/microcosm-cc/bluemonday/README.md

@@ -180,7 +180,7 @@ p.AllowElementsMatching(regex.MustCompile(`^my-element-`))
 
 Or add elements as a virtue of adding an attribute:
 ```go
-// Not the recommended pattern, see the recommendation on using .Matching() below
+// Note the recommended pattern, see the recommendation on using .Matching() below
 p.AllowAttrs("nowrap").OnElements("td", "th")
 ```
 
@@ -222,7 +222,7 @@ p.AllowElements("fieldset", "select", "option")
 
 Although it's possible to handle inline CSS using `AllowAttrs` with a `Matching` rule, writing a single monolithic regular expression to safely process all inline CSS which you wish to allow is not a trivial task.  Instead of attempting to do so, you can allow the `style` attribute on whichever element(s) you desire and use style policies to control and sanitize inline styles.
 
-It is suggested that you use `Matching` (with a suitable regular expression)
+It is strongly recommended that you use `Matching` (with a suitable regular expression)
 `MatchingEnum`, or `MatchingHandler` to ensure each style matches your needs,
 but default handlers are supplied for most widely used styles.
 
@@ -379,6 +379,8 @@ Both examples exhibit the same issue, they declare attributes but do not then sp
 
 We are not yet including any tools to help allow and sanitize CSS. Which means that unless you wish to do the heavy lifting in a single regular expression (inadvisable), **you should not allow the "style" attribute anywhere**.
 
+In the same theme, both `<script>` and `<style>` are considered harmful. These elements (and their content) will not be rendered by default, and require you to explicitly set `p.AllowUnsafe(true)`. You should be aware that allowing these elements defeats the purpose of using a HTML sanitizer as you would be explicitly allowing either JavaScript (and any plainly written XSS) and CSS (which can modify a DOM to insert JS), and additionally but limitations in this library mean it is not aware of whether HTML is validly structured and that can allow these elements to bypass some of the safety mechanisms built into the [WhatWG HTML parser standard](https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-inselect).
+
 It is not the job of bluemonday to fix your bad HTML, it is merely the job of bluemonday to prevent malicious HTML getting through. If you have mismatched HTML elements, or non-conforming nesting of elements, those will remain. But if you have well-structured HTML bluemonday will not break it.
 
 ## TODO

vendor/github.com/microcosm-cc/bluemonday/go.mod

@@ -3,7 +3,6 @@ module github.com/microcosm-cc/bluemonday
 go 1.16
 
 require (
-	github.com/asaskevich/govalidator v0.0.0-20210307081110-f21760c49a8d
 	github.com/aymerick/douceur v0.2.0
 	github.com/gorilla/css v1.0.0 // indirect
 	golang.org/x/net v0.0.0-20210614182718-04defd469f4e