memory regions

[mknyszek]

I'm starting this discussion to collect early feedback on a draft design for a kind of region-based memory management in Go. There is no prototype yet, only a design and a preliminary evaluation.

Please read everything below before replying, especially the design discussion section.

(Feel free to skip the detailed design, unless you're interested.)

Background

The arena experiment adds a package consisting of a single type to the standard library: Arena. This type allows one to allocate data structures into it directly, and allows early release of the arena memory in bulk. In other words, it adds a form of region-based memory management to Go. The implementation is memory-safe insofar as use-after-frees will never result in memory corruption, only a potential crash. Arenas have achieved real performance wins, almost entirely due to earlier memory reuse and staving off GC execution.

Unfortunately, the proposal to add arenas to the standard library is on indefinite hold due to the fact that they compose poorly with the language and standard library.

For example, builtin types all need special cases in the implementation, and require explicit slice- and map-related methods. Additionally, choosing to arena-allocate a variable means that it can never be stack-allocated, not without more complexity in the compiler.

Furthermore, for an API to make use of arenas, it must accept an additional argument: the arena to allocate into. There are far too many APIs that would need to be updated to make this integrate well with the language, and it would make those APIs worse.

The text below proposes a composable replacement for arenas in the form of user-defined goroutine-local memory regions.

Goals

First and foremost, our main goal is to reduce resource costs associated with the GC. If we can't achieve that, then this proposal is pointless.

The second most important goal is composability. Specifically:

• APIs should not need to be changed to take advantage of arena-like memory allocation patterns.
• Regions must compose with standard library features, like sync.Pool and unique.Handle.
• Regions must compose with existing optimizations, like stack allocation via escape analysis.

Finally, whatever we implement must be relatively easy to use and intuitive for intermediate-to-advanced Go developers. We must offer tools for discovering where regions might be worth it, and where they aren't working out.

Design

The core of this design revolves around a pair of functions that behave like annotations of function calls. It's useful to think of them as annotations, because crucially, they do not affect the correctness of code, bugs notwithstanding.

The annotations indicate whether the user expects most or all the memory allocated by some function call (and its callees) to stay local to that function (and its callees), and to be unreachable by the time that function returns. If these expectations hold, then that memory is eagerly reclaimed when the function returns, bypassing the garbage collector. If these expectations do not hold for some memory, then that memory is opted out of this early reclaim; management is passed on to the garbage collector as normal.

Below is the proposed new API which explains the semantics in more detail.

package region

// Do creates a new scope called a region, and calls f in
// that scope. The scope is destroyed when Do returns.
// 
// At the implementation's discretion, memory allocated by f
// and its callees may be implicitly bound to the region.
//
// Memory is automatically unbound from the region when it
// becomes reachable from another region, another goroutine,
// the caller of Do, its caller, or from any other memory not
// bound to this region.
//
// Any memory still bound to the region when it is destroyed is
// eagerly reclaimed by the runtime.
//
// This function exists to reduce resource costs by more
// effectively reusing memory, reducing pressure on the garbage
// collector.
// However, when used incorrectly, it may instead increase
// resource costs, as there is a cost to unbinding memory from
// the region.
// Always experiment and benchmark before committing to
// using a region.
//
// If Do is called within an active region, it creates a new one,
// and the old one is reinstated once f returns.
// However, memory cannot be rebound between regions.
// If memory created by an inner region is referenced by an outer
// region, it is not rebound to the outer region, but rather unbound
// completely.
// Memory created by an outer regions referenced by an inner region
// does not unbind anything, because the outer region always out-lives
// the inner region.
//
// Regions are local to the goroutines that create them,
// and do not propagate to newly created goroutines.
//
// Panics and calls to [runtime.Goexit] will destroy region
// scopes the same as if f returned, if they unwind past the
// call to Do.
func Do(f func())

// Ignore causes g and its callees to ignore the current
// region on the goroutine.
//
// Calling Ignore when not in an active region has no effect.
//
// The primary use-case for Ignore is to exclude memory that
// is known to outlive a region, to more effectively make use
// of regions. Using Ignore is less expensive than the unbinding
// process described in the documentation for [region.Do].
func Ignore(g func())

For some very basic examples, see the detailed design doc, or the next section.

Comparison with arenas

Where an arena might be used like...

func myFunc(buf []byte) error {
	a := arena.New()
	defer a.Free()

	data := new(MyBigComplexProto)
	if err := proto.UnmarshalOptions{Arena: a}.Unmarshal(buf, data); err != nil {
		return err
	}
	use(data)
}

... regions would be used like so:

func myFunc(buf []byte) error {
	var topLevelErr error
	region.Do(func() {
		data := new(MyBigComplexProto)
		if err := proto.Unmarshal(buf, data); err != nil {
			topLevelErr = err
			return
		}
		use(data)
	})
	return topLevelErr
}

You can think of a region as an implicit goroutine-local arena that lives for the duration of some function call. That goroutine-local arena is used for allocating all the memory needed by that function call and its callees (including maps, slices, structs, etc.). Thanks to some compiler and runtime magic (see below), if any of that memory would cause a use-after-free issue, it is automatically removed from the arena and handed off to the garbage collector instead.

In practice, we've found that the vast majority of arena uses tightly limit the arena's lifetime to that of a particular function, usually the one they are created in, like the example above. This fact suggests that regions will most likely be usable in most of the same circumstances as arenas.

Summary of benefits and costs

The core benefit is the potential for reduced GC overheads. An additional, more minor benefit is the potential for more efficient memory allocation. If the application code follows the region discipline, it makes much more sense to introduce a bump-pointer allocator for that memory (something like Immix; see the detailed design.

As alluded to in the previous section, some "magic" is required to dynamically escape memory from the region to the general heap. The magic is a goroutine-local write barrier (goroutine-local because it is only enabled on that goroutine, inside the region). We believe that we have a write barrier design that is cheap enough to make this worthwhile, incurring between 1–4% worst-case overhead when enabled globally, depending on the application (so it will be less in practice, limited to the goroutines that use it). We believe that this can be easily won back and then some in GC-heavy applications, provided their memory usage patterns line up with the region's assumptions.

However, this assumes that most or all memory in a region does not escape. The cost of promoting memory is higher, approximately the same cost as reallocating that memory on the heap (that is not how it would be implemented, but it gives you a sense of the cost).

Detailed design and implementation

For more details, please see the complete design document, which includes:

• A preliminary performance evaluation and cost/benefit analysis.
• Several proposed diagnostics to monitor use of regions.
• Alternatives considered.
• Prior art.

Detailed draft design.

(Note that the full design doc introduces a new term for memory 'escaping' a region ("fading") to avoid overloading with the compiler's 'escape analysis'. These mean the same thing.)

Design discussion

Below are a few discussion points that have come up often in early feedback, as well as my responses to those discussion points.

Goroutine-local region state seems problematic. Why is it OK?

Enabling region-based allocation for all variables created by a goroutine delivers a clear win if the vast majority of your memory allocated adheres to the region discipline. It's really OK if a small percentage (say, under 5%) of memory allocations escape from the region to the heap.

Also, note that the idea of implicitly opting-in memory was discarded for arenas, but that's because arenas can possibly introduce use-after-free crashes. If you use regions incorrectly, your program will not crash.

Will code owners need to consider applying region.Ignore everywhere?

One concern that was raised multiple times early in the design was whether region.Ignore would encourage tightly controlling allocations within regions so heavily that users would start pestering library owners to wrap certain portions of code in region.Ignore.

While this is something that could happen, I hope it would be rare, and I would encourage maintainers to push back on such requests if they occur. As mentioned in the previous discussion point, it's really OK if a small percentage of memory allocations escape from the region to the heap.

For example, I would explicitly advocate for not wrapping (*sync.Pool).New with region.Ignore in the standard library. Why? Because if you're using a Pool effectively, the number of steady-state allocations made should be quite small in practice, and easily overtaken by region allocations.

Given the concern however, perhaps we should remove region.Ignore from the design until we get more experience with it.

Possible extensions

Using PGO to automatically disable costly regions

If at compile time we see from a profile that the region slow paths are "hot" inside a particular region, the compiler can disable that region and potentially report that it did so. This technique has the potential to make monitoring more automatic.

Provide a GOEXPERIMENT to make every goroutine implicitly a region

This GOEXPERIMENT makes it easy to quickly turn regions on and off for an entire application. I suspect the majority of performance-sensitive Go applications, such as web services, would benefit from wrapping entire requests (usually captured by a single goroutine) in a region.

This idea is equivalent to enabling the request-oriented collector, an experimental garbage collector from early in Go's life, designed by Rick Hudson and Austin Clements. The difference between that design and this one is in the details: separately managed memory, and a much cheaper write barrier fast path.

This may also combine well with dynamically disabling regions with PGO.

Provide a GODEBUG to disable all regions

This allows for quicker rollback and experimentation. We can also extend this GODEBUG to work with compile-time hash bisection to identify costly regions efficiently. This is made possible due to the fact that regions do not change the semantics of the program.

Next steps

Although fairly fleshed out, this design does not yet have a prototype. Before making such an investment, we wish to gauge interest from the community.

Once we feel that broad interest exists, we may prioritize it. This would then involve building a prototype, available as a GOEXPERIMENT, which would then be used to steer the design, possibly enough toward approval. Note that we plan to remove arenas from the standard library once this prototype is created.

[gsymons]

why not go:region directive

//go:region
func myFunc(buf []byte) error {
    data := new(MyBigComplexProto)
    if err := proto.Unmarshal(buf, data); err != nil {
	    return err
    }
    use(data)
}

[gsymons]

Hi @gsymons, that would make it effectively an attribute of the function definition?

As I understand, part of the goal of the design is to make it a caller decision rather than being an attribute of the function itself. In your case, annotating the function might then make it harder to then use that annotated function without a region?

OK, caller decision is important if the feature has side-effect. Nonetheless, if the region functionality becomes widespread, I suggest we investigate other design patterns to minimize the use of closures.

[SerJaimeLannister]

Hey , I am really really sorry that I couldn't understand what this is supposed to bring to the table.
Could someone please explain it to me like I am 5 / rookie go developer (because that's what I currently am)

[rferreira]

Hey , I am really really sorry that I couldn't understand what this is supposed to bring to the table. Could someone please explain it to me like I am 5 / rookie go developer (because that's what I currently am)

It provides deterministic memory control, so you can control deallocation and not the runtime GC. Not something you would use normally.

[SerJaimeLannister]

Hey , I am really really sorry that I couldn't understand what this is supposed to bring to the table. Could someone please explain it to me like I am 5 / rookie go developer (because that's what I currently am)

It provides deterministic memory control, so you can control deallocation and not the runtime GC. Not something you would use normally.

does this allow golang to be faster for some use cases ? Like I might write code in golang and if its not fast enough then I can use these memory regions to drop memory count ? what are its side effects / lose ?

[ianlancetaylor]

As the description says, that goal of this proposal is to allow programmers to reduce the resource costs associated with the garbage collector. In particular, if your program handles requests with clearly bounded memory requirements, as is true of many servers, and if your program has significant garbage collection costs, as is true of some servers, then the hope is that careful use of regions may reduce the garbage collection costs.

[ydnar]

Provide a GOEXPERIMENT to make every goroutine implicitly a region

I think this is worth considering in isolation.

[robaho]

I don’t like it at all. Arena memory management work great in C++ because you are already manually managing memory. Bringing that to go is a step backwards. Better to integrate different GC algorithms that better handle some workloads.

Adding more manual memory management to Go is a bad choice. Already I guess that unsafe causing 90+% of all of Go hard to debug bugs.

[robaho]

The difference is that these are GC backed native buffers with limited usefulness. I wouldn’t use new age Java as a great reference - a lot of the stewards have thrown in the towel and it’s become a kitchen sink language. It’s doubtful many of these features will survive and they’ll be deprecated for low adoption.

Still these are designed more for FFI than as a performance optimization. It is doubtful they’ll perform better than using heap memory.

[SerJaimeLannister]

Still these are designed more for FFI than as a performance optimization. It is doubtful they’ll perform better than using heap memory.

Sure I agree but just because something in java doesn't make it bad even if you hate it. Its a collection of those things combined in that specific time .

Here Java probably wasn't talked about as great reference in the language is the best but rather that its a mature language.

Hate it or love it , though I am not a java enjoyer. I have heard good things about modern java being a beast of its own by many admirable people and you may dislike this opinion and that's okay but still your criticism was rather shallow and I deem that we should atleast appreciate efforts of somebody putting efforts into our favourite language instead of jumping to conclusions.

[robaho]

Not sure what you’re referring to. I am a proponent of Java - I’m against some recent kitchen sink type changes - others like Virtual Threads have been game changers. It’s more the pace of change without enough review has been a problem.

As to Go, I’m also a fan, which is why I think this is a bad change as proposed. Better to add a generational collector and do this transparently.

[robaho]

Better to integrate different GC algorithms that better handle some workloads.

That is effectively what this is.

Already I guess that unsafe causing 90+% of all of Go hard to debug bugs.

The proposed mechanism is safe.

Right, you are having to manually control the GC. Don't get me wrong, it is a novel way of getting around Go not having a moving/copying/generational collector, but it is a step backwards for Go in my opinion.

I also never said it was unsafe. I was comparing it to the manual effort required in using unsafe, and the problems that arise from that. I can see similar cases here where regions, and nested regions in libraries cause a performance degradation . Maybe I am wrong.

Just seems to me that if the developer is "smart enough" to create a region, the compiler/runtime should be even smarter in this area, and do it automatically.

[ianlancetaylor]

Doing automatic region allocation was the goal of the request oriented collector mentioned in the proposal. Unfortunately after considerable work it was not an improvement.

As the proposal says, we could turn regions on automatically using a GOEXPERIMENT if it proves worthwhile.

Go already provides considerable manual control in memory allocation. That is what makes it hard to do better automatically--the programmer has already taken care of the low-hanging fruit.

[Danlock]

I am in favor of removing Ignore(), as personally it seems so much simpler to reason about programs using this feature without it.

What happens when a CGo call happens in the middle of region.Do?

[aep]

I want to agree, the use case for that seems optimization, since the proposal would already ignore variables that have to escape to heap automatically. code that wants to explicitly optimize use of memory lifetime on the heap should just use a pool instead.

but i'm not sure if I read the reasoning for pool correctly. if this is also part of the region, it doesnt help.
Developers need some way of micromanaging explicit heap allocation because there's definitely corner cases where the automatic escape is too expensive.

I just wish ignore was less conveniently available. in most cases pgo should optimize this better than micromanaging.

func fetchPicture(id) []byte {
  if !cache[id] {
     // this should not be bound to the lifetime of the caller
     cache[id] = largeAllocation
  }
  return cache[id]
}

i'd feel more comfortable with ignore if it was explicit for just one allocation. like heap.new(BigThing) or something.
or just dont. we already have closures thanks to goros that can be abused to do explicit lifetime binding:

func init() {
    go region.Do(cacher)
}
func cacher() {
   for range request {  allocate cached mem here }
}

func fetchPicture(id) []byte {
   make channel
   <- etc etc
}

[mknyszek]

A cgo call in a region.Do that returns back to Go is fine from a correctness standpoint, but may have other costs in some specific cases. It's not like C cares -- it can't allocate Go memory anyway. Plus, you can't write Go pointers into C memory that stay there beyond the C call unless they're explicitly pinned, and so explicit pinning would require unbinding memory from a region.

However, we could probably make an exception for memory that is pinned for the duration of the cgo call (the way passing values through cgo is currently defined to work). That could, I think, safely stay inside the region, and the region just stays active across the cgo call.

One thing I haven't given a lot of thought to is what happens if Go calls into C, which then calls back into Go. Should the region still be active? I'm not sure, I'll give it more thought.

[qiulaidongfeng]

I wonder if this or a similar scheme could be used in the future for situations where multiple goroutine objects have the same life cycle?
My use case is a compiler that I wrote myself for the purpose of researching and learning compilation principles.
In https://gitee.com/u-language/u-language/blob/master/ucom/main.go#L258 , if it exec to https://gitee.com/u-language/u-language/blob/master/ucom/parser/parser.go#L51 , exec to https://gitee.com/u-language/u-language/blob/master/ucom/parser/buildmode2.go#L63
The abstract syntax trees of multiple files are parsed in different Goroutines and have the same lifetime.
Having the compiler and gc manage memory for such use cases with known lifecycles may not seem like a good solution. Some problems such as #68815 #68974

[perj]

I'm slightly concerned about the use of the stdlib package namespace. I reacted already at the weak package, which also seem very small in scope. The region.Do seem to work just as well as runtime.Region.

Identifiers already often conflict with stdlib package names, most commonly path. Region seems like another one that will conflict fairly often.

[mknyszek]

Personally, I don't care too much what the name is or where the API lives. Of course it matters, but I feel like that's something to iterate on in the actual proposal process if we make it there.

(The main reason this lives in its own package is because of Ignore. If Ignore is dropped, there isn't a very good reason to define a new package for a single function, IMO.)

[dpifke]

Ignore could be runtime.IgnoreRegion or runtime.RegionIgnore.

Agree that the package name "region" is likely to conflict with existing code all over the place. There isn't a great abbreviation for it (rgn? reg? r?), so having to refactor code to deconflict is likely to make it less readable.

[seankhliao]

from golab: region.Do and region.Dont

[mikeschinkel]

@seankhliao — My inner Yoda would prefer region.DoNot, especially given that in Go there is no try. 😁

[jellevandenhooff]

Is there any guidance on package names for the standard library? The ship has sailed for unique and weak, but in retrospect I wish they all lived in a package together. Something like gc.UniqueHandle, gc.WeakPointer, etc. could have been nice.

[Splizard]

Can't the compiler do escape analysis at the goroutine-level and use this as a hueristic for guiding such an allocation optimisation? Wouldn't all self contained goroutines benefit from this? I don't see any reason why a new API needs to be introduced for this.

It would be disappointing to see yet another interesting technical optimisation pushed unnecessarily into the language through some new magical standard library package.

[Splizard]

@ianlancetaylor

For example, slice data will always escape unless the compiler can somehow calculate an upper bound on the length of the slice

This reply thread is very particularly focused on allocating memory within goroutine-local memory arenas/regions, not the stack. Slices that do not escape their goroutine would be allocated here as they escape from the stack. As such, I do feel the need to reject your assessment that this is an entirely independent topic. The question I've raised here is, why does this need to be exposed through a new compiler-package versus the use of static analysis techniques?

We've been taking a closer look at interfaces and by extension closures here as they appear to be the primary hurdle here.

you are saying things like "the compiler is aware of all interface implementations at compile-time" which is simply incorrect.

This is intended to convey (setting aside plugins) , that a Go compiler toolchain as a whole (including the linker, the type checker etc) has access to all named types within the source of a Go program as well as all interface types and could compute the set of types within a program that implement each interface.

I don't see it being meaningful nor useful to interpret this as "the current standard Go compiler by the Go team knows the concrete type of every interface at every call site ahead of time for any Go program", which is obviously not correct, nor possible.

there simply isn't anything in the toolchain to support this kind of analysis

Since we've established that this is possible and that the primary concern with doing this entirely ahead of time is too difficult and/or would incurr an unacceptable hit to compile speed. My next question would be, why not partially determine this at runtime?

If escape analysis were extended to goroutines (in the manner that @randall77 mentioned) and this information is recorded for each argument of each interface method and closure then this can be queried at runtime before dispatch to determine whether an argument needs to be unbound/moved-out from the region. In some cases the memory can be allocated in the appropriate location to begin with.

I would imagine that in real world Go code many arguments to functions can be proven not to escape their goroutine.

[atdiar]

@Splizard I guess the question is what if a method conditionally assigns a value (allocated in the region) to a global object? [edit] an object with an interior pointer pointing into a region that is, for instance.

What if the interface being implemented has its implementing value depend itself on some codepath known at runtime?

What kind of conditional paths could someone be applying?

Whether it is setting something at the linker level via ldflags or even at runtime.

I'm wondering if a static analysis might not become too blurry due to compounding false positives.

That's when some help is typically required, often under the form of annotations or other mechanisms.

I believe you're right that whole program wise, we know some things. But escape analysis is still flow sensitive if we want to be somewhat precise.

That's why there are advantages in relying on nudges/intent from devs, explicitly declaring regions: it should be verifiable.

[ianlancetaylor]

@Splizard Thanks for the note.

I apologize for misunderstanding what you meant by "compiler." That said, words do matter. When you say "compiler," I think you mean the compiler. I don't think you mean the entire toolchain.

This is particularly the case because doing whole-program analysis at link time, especially if we expect to use that analysis to change the generated code, is not something that we are going to do.

I take your point that we can do a kind of escape analysis that allocates goroutine-local memory for slices that do not escape.

I agree that in some cases it may be possible to get escape information dynamically, and generate different code paths based on that information. That said, some of this information is difficult to calculate, even dynamically. For example, arguments passed to fmt.Printf always escape, because fmt.Printf may call the String method, and that may cause the value to escape. To represent that even dynamically is difficult, because in general the call to fmt.Printf may itself be passing interface values whose static types are unknown. So while in general the compiler can in some cases determine that the arguments to fmt.Printf don't escape, it's not obvious to me that this can be done without writing special cases for fmt.Printf. And of course we don't want to write special cases for everything that comes up. For another example, does the buffer passed to io.CopyBuffer escape? It depends on the ReadFrom and WriteTo methods.

So I have come around to agreeing that there are some possibilities here. But I don't think it's clear that we should pursue those possibilities, which may not pan out, in favor of pursuing the memory region proposal.

[Splizard]

@ianlancetaylor
Cheers, I appreciate your response. Here's the thing, I don't think you need to do whole-program analysis at link time, let's say for each argument provided to each local function, each compilation artifact records "Does argument X of function/method A escape the stack/arena?", take the fmt.Printf may call String example, each (T) String() string, method signature can report this information into the compilation artifact at a fixed addressable location. Then at link time, you AND these values together into a final global addressable value and use this result at runtime as part of the decision on where to allocate. So any place you call fmt.Printf the arguments can be allocated on the stack/arena dependant on this global value. When a plugin is loaded, it ANDs its own values together into the host runtime.

fmt.Stringer and io.Writer interfaces are great examples of where in practise they don't escape their arguments and if they do then this is something tooling should clearly report so that it can be resolved. I imagine most interfaces fit this profile. In other words, either each argument is expected to be able to escape due to the nature of their intended behaviour, or they should never escape. You could do the same thing with closures but perhaps to a less reliable effect.

For the io.CopyBuffer example, where buf is allocated can be decided upon by the logical AND of the addressable escape values recorded for buffer arguments for both the ReadFrom and WriteTo methods.

Does this help to clarify whether or not this would be worth pursuing? I've never said don't do memory regions, I'm saying why not drop the API and use escape analysis (in this case supported at runtime to reduce compilation time and to support dynamic loading) to make these sorts of allocation decisions of whether to allocate on the stack, a goroutine-scoped arena or the heap.

[ianlancetaylor]

A lot of analysis is required to work out that the reason arguments to fmt.Printf escape is because they have a String method. I wouldn't assume that is feasible without some clear understanding of how to do it.

Using a global to record whether memory escapes in simpler cases is possible but means that we have to compile two code paths in many different cases. That will slow down the compiler and increase program size. I would expect that often one of the code paths will be taken very rarely, so it won't get much testing.

So I'm skeptical that the substantial effort required will pay off.

[zephyrtronium]

What happens with variables that internally own memory? Are there any differences in what memory is bound to the region between these examples?

// Example A: No region.
x := big.NewInt(1)
for range rand.IntN(32) {
    x.Lsh(x, 64)
}
return x

// Example B: Region for the loop.
x := big.NewInt(1)
region.Do(func() {
    for range rand.IntN(32) {
        x.Lsh(x, 64)
    }
})
return x

// Example C: Region per iteration.
x := big.NewInt(1)
for range rand.IntN(32) {
    region.Do(func() {
        x.Lsh(x, 64)
    })
}
return x

// Example D: Explicit copy out of the region.
x := new(big.Int)
region.Do(func() {
    y := big.NewInt(1)
    for range rand.IntN(32) {
        y.Lsh(y, 64)
    }
    x.Set(y)
})
return x

[mknyszek]

Values that internally own memory created outside of a region are very likely to have their allocations unbound, since their internal memory needs to be reachable from outside the region (the original value, in this case, the big.Int).

More specifically, since B and C are mutating an outside-of-region structure inside the region, then any internal allocations would be unbound from the region immediately. In this way, A, B, and C are all approximately the same, but with B and C you're paying an extra cost to unbind from the region.

With D, you only unbind the last value, which means all the intermediate allocations get to be cleaned up.

[timothy-king]

More specifically, since B and C are mutating an outside-of-region structure inside the region, then any internal allocations would be unbound from the region immediately.

@mknyszek Can you say more about how this is done/what "unbound from the region immediately" means? For simplicity lets assume x has type *struct{f []uint}. We come in with x and x.f allocated from some outer region Q from big.NewInt(1). In example B, we enter a new region R on region.Do, and then we call x.Lsh(...). That gets to some internal x.f = make([]uint, ...) line. Does something different to today's compiler/runtime happen at the point make is called due to regions? Does the unbinding happen when R is exited?

My questions were answered in the detailed Implementation section.

[CannibalVox]

For arenas and now regions, the obvious use case is allocating request and response objects. This has far better ergonomics than arenas, so that's very good, but the design seems to be fooling itself a bit: obviously most go programs in the wild would have to live entirely inside regions if this plan went forward, since generally speaking 100% of all running code will be an in flight endpoint call that needs access to the request that initiated it.

Making all goroutines have regions isn't just an interesting experiment, that is an accurate description of how most production go code will run and if go can't handle it, this proposal cannot succeed.

I've mentioned it in the past, and I know language changes are a serious issue, but one possibility not explored in the design doc is a type keyword that would prevent taking actions with a variable that would result in its data being promoted out of its region (enforced by compiler), thereby allowing region-allocated data to be safely used read-only on the stack within an ignore goroutine.

It's a big lift, but if it doesn't exist then everything is going to be in a region anyway and this proposal should probably acknowledge that.

[marcintustin]

In practice I think whatever proposal succeeds needs to allow memory to be shared with child goroutines, and so be scoped to a context rather than a goroutine. Or alternatively explicitly support spawning a goroutine attached to an existing region.

[atdiar]

I believe the mechanism is taking this into account already. As soon as there is shared memory between goroutines, we can automatically opt-out of region-based memory handling in favor of traditional memory handling.

If this feature is stuck in PGO-land, that means that traditional is always the default and establishing a region is a runtime concern.
Not sure why leaving it to the runtime though.
On the other hand, is it valuable for every goroutine to automatically establish a region or is it better to leave it as a dev concern? The design docs hints toward the latter since fading is control-flow sensitive.
Otherwise, it could definitely be a compiler optimization pass on every goroutine of a program.

[robaho]

I am not certain why you couldn’t get most of the benefit transparently with a small object non copying per routine generational collector/region. So objects that are recently allocated can be quickly collected - hopefully cleaning the region. The region could also support bump allocations if mostly empty - which is the expectation of a region biased workload.

[mknyszek]

Connecting this proposal to generational approaches is reasonable. But generational garbage collection has substantial costs that often don't pan out for Go programs. We have tried it, in at least two different iterations.

In short, some applications benefit. Many do not.

At least part of the problem appears to be that, in practical terms, the weak generational hypothesis does not hold up well with respect to heap-allocated memory in many real-world Go programs (think 60-70% young object mortality vs. the 95% typically expected). If the weak generational hypothesis does not hold up well, we're not reaping enough memory each minor cycle to cover the global fixed costs: you will have more GC cycles, so those GC cycles better be a good bang for your buck. In particular, generational collectors need some way to track old->new pointers. Even a simple card marking write barrier is relatively expensive because it is always enabled.

So why does the weak generational hypothesis not hold up well? Compared to Java, many fewer objects get allocated on the heap in Go due to compile-time escape analysis. Empirically, for the same CPU burn, I've observed about a 5x difference. (From this perspective you could argue the weak generational hypothesis does hold, and you might be right. Let's just focus on heap memory though, because we're talking about generational garbage collection techniques.) Furthermore, many more objects are likely to out-live a GC cycle because GOGC=100 is somewhat aggressive as a default. JVMs typically use more memory by default and have longer cycles. And defaults matter.

We have put additional thought into this over the years. There may yet be a way to do it and make it widely profitable, but if so, it's not obvious. It is also possible we made mistakes in our previous attempts and they could be rectified, but it's hard to justify the complexity of a global generational mechanism with the risk of many real programs performing less efficiently out of the box. (Could it be a GOEXPERIMENT like the design discussion here suggests? Sure. But generational garbage collection is still global in a way that this isn't and so you lose some nice locality properties. These properties allow one to, for example, disable regions with PGO. But you can't just turn off the costs of generational garbage collection in such a localized way.)

Part of the reasoning behind this proposal, and the reason this proposal is an API, is to provide a localized mechanism for opting into something sort of generational-like where appropriate. The core observation here is that bounding lifetimes of most objects in most programs is not too hard for someone to do, and they can profit. We learned this from the arena experiment.

To respond to your other concern in a different thread about cognitive overhead: this API is somewhat niche. The point is to apply it to the performance-sensitive part of your code when you have something real to gain. Most Go programs likely do not care. But some will, namely the resource-sensitive and latency-sensitive ones. Once you're already incurring the cognitive overhead of this sensitivity, my guess is that the additional cognitive overhead of regions is a small delta. And given that it can be somewhat locally reasoned about, the hope is that the cognitive overhead is also localized, roughly the same way it would be if you're optimizing a particular codepath.

I hope this is helpful in providing some context on how this specific design came to be.

[robaho]

I confess that I am unfamiliar with the proposed implementation, but I suspect that that since others have stated this cannot be done statically at compile time, then at the moment an object must be allocated on the heap (which is determined at compile time in Go), it must decide that a region is in effect, and allocate in the region, then using a mechanism I am not clear about, if that object is then passed to another Go routine, or is made a reference of an object outside of the region, it must be moved from the region to the global heap and all pointers to the object updated. Is this correct?

edit: if the object is made an element of another object that escapes the region, then this object as well must be moved.

[jellevandenhooff]

@robaho when a pointer to a on object allocated in the region is written outside of the region, the object is not moved, but marked as “faded”. When the region is later released, the faded objects are kept alive. The region in that way is different from an arena: all the non-faded allocations are freed, but not the entire arena. This leads to (some) memory fragmentation, but the proposal argues that fragmentation cost will be limited.

Freeing the non-faded allocations is faster than a normal GC cycle because the runtime knows that any non-faded objects have no references outside the region and can be released without further marking and sweeping.

[robaho]

Interesting. Thanks for the clarification. This has been tested - because on the surface it seems the fragmentation would be excessive unless the runtime marks the variable/code as fading for all future allocations - which I assume is stored in a map in the region object? Doesn’t that mean that every allocation needs a check against the map - so it is not simply a bump allocation?

[marcintustin]

Why not tie the lifetime of the region to a context? Then escape analysis from the goroutine would be unnecessary AND this could be used across goroutines. Specifically this would make it usable for the whole lifetime of an http request.

If we were getting really fancy, we could add a new variant of new, newCtx(ctx, type, size…) that uses the allocator associated with a context, or the default global allocator if none is associated with the context.

[marcintustin]

Ah maybe I should make my own proposal :) I’m ok with an opt in mechanism that can cause crashes.

[robaho]

The problem with the crash proposal is that it affects the stdlib and others libraries that aren’t aware they’re running in an arena.

[marcintustin]

What’s a scenario where the Stdlib would fall foul of this if access is scoped to a context or goroutine tree, that the caller couldn’t anticipate when opting that part of their program into regionalised memory management? I view it as being the same as passing a nil pointer to the stdlib - you expect that to panic.

[robaho]

It allocates an object and stores it in a preexisting cache map. When the arena is cleared the cache now contains an invalid pointer.

[marcintustin]

Ah yes either the stdlib would have to use only regionalised pools, or programs would have to be able to request creation of memory in the global region to be able to send it to libraries using global pools.

Tbh I think being able to request allocation in the global region would probably be desirable, but an api that requires the programmer to keep track of which region objects are in would probably be more fiddly than we like in golang.

[ivankorobkov]

Hi, thank you for your detailed proposal and explanations.

Could you answer some questions, please?

1. Why not make every goroutine region-based by default?

2. What happens when a region-based object is passed to an outside function/closure, which can outlive the region?

func myFunc(buf []byte, fn handleFunc) (err error) {
	region.Do(func() {
		data := new(MyBigComplexProto)
		if err = proto.Unmarshal(buf, data); err != nil {
			return
		}
		fn(data) // data passed to the function outside of the region
	})
	return
}

3. What happens when a region-based object is passed to another goroutine?

func myFunc(buf []byte, fn handleFunc) (err error) {
	region.Do(func() {
		data := new(MyBigComplexProto)
		if err = proto.Unmarshal(buf, data); err != nil {
			return
		}
		go process(data) // data passed to another goroutine
	})
	return
}

Thanks

[Merovius]

From other replies in this thread:

1. Because there is a cost and it is likely not worth it, in all cases. In particular, this would be basically equivalent to the "Request Oriented Collector" tried out a few years back, which sped up some programs, but slowed down other and was not a clear win.
2. Regions are stack-specific, so AIUI in your example fn is actually inside the region. If fn escapes data (e.g. by writing it to a global variable), then the underlying array of data would be marked as faded and not collected with the rest of the memory in the region.
3. Same.

[prymitive]

1. Because there is a cost and it is likely not worth it, in all cases.

I wonder how does someone arrive at a conclusion that some code would benefit from a region. Is the only option to try it really? I think the danger is that a lot of people will just wrap lots of code in Do() hoping it does something good.

[robaho]

is that a lot of people will just wrap lots of code...

This is exactly what I was hoping to avoid. Or what is more likely, is that someone does all of the analysis and figures out the optimum place for a region.Do() and several code changes later by people other than the original author invalidates this. So on every code change, you have to understand the relationship with a potential region that may be several layers higher. This is brittle and troublesome - it is essentially manual memory management (although in this case it is safe, so you will only have a potential performance penalty). I am more concerned about all of the additional programmer effort it is going to take to document all of the assumptions and criteria for why a particular piece of code was wrapped in a Do() and then all of the additional effort of a developer to ensure their code changes adhere to the same constraints. Brutal.

This is why I think it must be transparent - e.g. an outcome of PGO - or it shouldn't be added.

Sometimes just because a change may help a few people in a few scenarios, doesn't mean it should be exposed to the world.

[mknyszek]

I wonder how does someone arrive at a conclusion that some code would benefit from a region. Is the only option to try it really? I think the danger is that a lot of people will just wrap lots of code in Do() hoping it does something good.

That's definitely not my expectation.

My expectation is the people that care about performance, that will even bother to reach for this, are already thinking about how their programs are organized. I believe that, in many cases, developers can reason about their code and make a reasonable inference. This belief is not founded upon nothing -- we provided virtually no tools for deciding when arenas would work well, and yet they were used effectively. It's usually pretty clear when a whole bunch of allocated memory has nothing to do with any other goroutine, and doesn't get returned. When it's not clear, we can provide some tooling to help identify it (the detailed design doc has ideas). But again, it's exactly how people inferred good places to use arenas, but without having to update quite as much code to actually try it out.

I am really not very worried about misuse of regions. I think it's true of most features that blindly using them is likely to yield poor results.

Or what is more likely, is that someone does all of the analysis and figures out the optimum place for a region.Do() and several code changes later by people other than the original author invalidates this. So on every code change, you have to understand the relationship with a potential region that may be several layers higher. This is brittle and troublesome - it is essentially manual memory management (although in this case it is safe, so you will only have a potential performance penalty). I am more concerned about all of the additional programmer effort it is going to take to document all of the assumptions and criteria for why a particular piece of code was wrapped in a Do() and then all of the additional effort of a developer to ensure their code changes adhere to the same constraints. Brutal.

I don't think there's much evidence that regions are going to be brittle in practice (the opposite actually; that was a big focus of the preliminary evaluation).

I also don't think regions are comparable to manually managing memory. Manually managing memory has a risk of crashing or memory corruption. At worst, this is like carefully tuning your code to stack-allocate memory when possible, which people already do successfully, and that's far less forgiving and far more brittle than what's proposed here.

[glycerine]

Thanks for working on this. It sounds fantastic.

Allowing regions to span goroutines dramatically complicates their implementation, enough that it may not actually be a profitable feature anymore.

This comment (above) helps my mental picture of how regions would work.

Question: As I potential user of regions, one question in my mind is: do I need to concern myself with the (region or heap) origin of the memory (a pointer to struct of some kind) that I send on a channel? Is there an efficiency difference if something is coming from a region because it needs to be copied to the garbage collected "region" so to speak?

Comment: a nice side effect of any design would be to enable users to experiment readily with memory allocation strategies, to find those most appropriate for their code. I wrote an off-heap hash table for my Go code about 10 years ago, and the main pain-point in use was that I had to serialize and deserialize everything manually to move objects between the heap and the manually managed memory ( https://github.com/glycerine/offheap ). Ideally I imagine that region use could be like the net/http package having a default mux: Having region.Do() and region.Ignore() operate on a default memory manager, but a specific instance could also be instantiated and tweaked. I could then call Do() on my specific instance to get allocations to (in my example) my own previously malloc-ed region of memory that is off heap; writing into my hash table there; ideally without having to manually do the serialization, nor having to chase pointers to child objects to get transitively completely self-contained objects.

Beside the off-heap hash table, a second use case for user-customization would be when running in WASM code on a web-worker thread, and wanting to pass memory to a WasmGC implementation for other components to use. Go compiled to Wasm is probably never going to be able to use WasmGC's moving collector, because it is moving and lacks support for interior pointers, but we might well want to be able to inter-operate with other languages that do.

[ianlancetaylor]

In this proposal nothing gets copied into a region. Objects are allocated in a region, or they are not. Once they are allocated, they do not move.

When receiving an object on a channel, you don't need to worry about where it was allocated, except that 1) if it was allocated in a region, it will escape that region, so maybe don't do that; 2) I am ignoring NUMA issues.

[andrewbaptist]

Thank you for the work on this! I'm excited to see this completed.

Regarding the concern in the design related to allowing regions to cover multiple goroutines, this might dovetail really well with a structured/scoped concurrency approaches like https://github.com/sourcegraph/conc. If the lifetime of a goroutine is entirely within a region.Do, then the API issue because tractable. Unfortunately that project hasn't had much activity lately.

In terms of the implementation issue, I agree that it might be better to leave this as a "potential future enhancement". It would be reasonable for this to be higher cost and possibly even have a variant which allowed the user to specify the "low cost single goroutine" region or the "higher cost including child goroutine" regions. Something like region.Do and region.DoWithChildren.

The paradigm I'm thinking about is something like a request/response microservice that kicks off goroutines to make additional requests along the way. Everything is bound to the top level request and when that is complete the entire request and all sub-requests will be destroyed. Within the requests there could be data sharing through channels or other mechanisms, but it would all be in a single region.

[robaho]

This doesn't seem to jive with Ian's comments that Go programs don't adhere to the weak generational hypothesis - because it seems all that would entail is a generational collector, or the request oriented collector would have shown better results https://docs.google.com/document/d/1gCsFxXamW8RRvOe5hECz98Ftk-tcRRJcDFANj2VwCB0/edit?pli=1&tab=t.0

[mknyszek]

The request-oriented collector would've still struggled with this, because it's goroutine-oriented. This may be a case where generational garbage collection helps if reclamation even works out. (For example, if your RPCs/tasks are long and exceed 1 GC cycle, you're likely to not benefit from generational collection.)

FTR, you are not the only one to suggest you would benefit from scoping regions across multiple goroutines. The costs would likely be global, and would likely be similar to generational garbage collection, so I think if we were to go down this road, thinking about that again might be worthwhile. But, a region-style thing still might be worthwhile over generational garbage collection if the semantics were more like the Yak GC (see the detailed design doc).

You're also right that it could (probably) be implemented as an extension to this proposal.

[jellevandenhooff]

Bit of a crazy idea: What about a function region.NoUnbind(f func()) which guarantees memory allocated will not be unbound. Any memory allocated inside of f gets marked with a special bit, say "colorful", which is not allowed to be faded. The write barrier could check the "colorful" and panic when it attempts to fade the allocation.

Probably not worth adding, but curious how you imagine thinking about debugging and/or instrumenting code using regions.

[robaho]

I don’t know. Different languages are designed for different purposes and if you try to design a language that work well for any task it is usually a failure. Java & Python gives up performance for safety and ease of use. C++ and Rust give up ease of use for performance and small binaries.

Go sits in the middle of these and probably always will imo - and I don’t think that is a bad thing - it’s actually pretty great.

[CannibalVox]

Java & Python gives up performance for safety and ease of use

These two languages are in totally different performance classes, and go is generally slower than one of them due to the exact issues that I'm describing.

[robaho]

My testing doesn’t bear that out for many classes of applications. As I mentioned check out the performance table at github.com/robaho/cpp_leveldb in real world settings.

even if you are correct, this will actually make the problem worse I believe with the additional write barrier. If lowers the GC cost not the allocation cost. I plan on reading the detailed design doc today.

[thepudds]

I also recall a brief mention of the possibility that arenas may be used internally at google even after the change was shelved though obviously I have no idea whether that happened.

FWIW, the detailed design doc does reference the use of the arena experiment at Google:

In practice, all uses of an arena within Google tightly limit the arena's lifetime to that of a particular function, usually the one they are created in, like the example above. This fact suggests that regions will most likely be usable in most or all of the same circumstances as arenas.

In other words, the suggested region experiment is building upon learnings from prior experiments and not entirely just theoretical speculation. (And the fact that several of those prior experiments have been set aside might give some reassurance that the Go team would likely be willing to set this experiment aside as well if the performance results aren't good enough as happened with the Request Oriented Collector, or if the code impact is anticipated to be too large like with the arena experiment, or if otherwise determined to be not a good fit for Go.)

[atdiar]

@CannibalVox @robaho I think it should be a wait and see.
The design docs explains that allocating can be expected to be faster.
In fact, it would make sense since we are dealing with coarser spans of memory with potential patterns of recycling.

I can already think of ways this can be very useful, for instance when writing lock free ring buffers (quite used in java world in hft for instance). Mainly to communicate intraprocess (or is it interprocess? Basically bridges).
That could be a boon for wasm and/or interop with other runtimes (in my case I'm thinking swift and android).
These are just ideas. It might or might not pan out exactly. Still worth exploring.

But the way this proposal appears not too intrusive language wise, I have good hopes personally.

[objectref]

I am really excited about this proposal! Giving extra power to the language (when you really need it) with minimal programmers’ effort is something I would really like to have and it fits really nice with the spirit of Go.
I hope it will be accepted and move forward.

[glycerine]

I was surprised at the last word in this sentence from the detailed design. Was meant to be "regular heap spaces"? Is there such a thing as a "regular heapArena"? It feels like it might be a typo.

Address Space
When these are created, we provide hints to mmap and the like to be contiguous and in a distinct part of the address space from the heap, though it's OK if these special region heapArenas interleave with regular heapArenas.

[thepudds]

FWIW, the runtime already uses heapArenas internally:

go/src/runtime/mheap.go

Line 238 in f1add18

// A heapArena stores metadata for a heap arena. heapArenas are stored

[mknyszek]

A heapArena is an existing structure in the runtime. So what I mean here by "regular heapArena" is just one without its corresponding isRegionArena bit set.

[glycerine]

For example, the compiler‘s escape analysis may be able to deduce that some value may be stack allocated, but this analysis immediately fails if the value is allocated into an arena.

This (detailed design doc) sentence asserts surprising a claim without any backing rationale. If the traditional escape analysis says something can be stack allocated, how is it possible that, all of a sudden, it now cannot be stack allocated if it is under a region.Do() call?

[mknyszek]

That's talking about arenas (#51317), not region.Do.

[glycerine]

That's talking about arenas (#51317), not region.Do.

Ah yes, I missed that. Still, a short reason why arenas collide with escape analysis would help me understand why it doesn't apply to the implicit design as well.

I would like to gently suggest that clarity could be improved by adding a concluding, final sentence to that paragraph, along the lines of, "In contrast, our implicit region design is able to continue to leverage the existing compiler's escape analysis for efficient stack allocation."; if such a claim can be truthfully made.

[Merovius]

@glycerine Stack allocation has to be reserved by the compiler. Arena allocation was an API call. So the reason that arenas don't play nice with escape analysis is that with arenas, the code explicitly says "please put this on (a special part of) the heap". It's the implicitness of the region design that make it work with escape analysis.

[glycerine]

Thanks Axel.

[atdiar]

@glycerine and also because arenas require manual explicit freeing.
So a value allocated in an arena should be live as long as the arena exists. Escape analysis doesn't directly check the arena bounds.

Unlike this region-based design that uses function scopes to define lifetimes (hence the relation to escape analysis although it's renamed fading to be more distinct)

[glycerine]

The detailed design doc was very helpful in clarifying for me one main point.

This is not a region design.

I like it, and I think its great. I just think it is mis-named.

Arenas and regions are synonyms, and this design is neither. It is an enhancement to the garbage collector based on a user annotation or hint.

So my primary feedback is about the writing and presentation, not the merits of the design.
The design seems fabulous and well worth doing.

But for the presentation: I think calling this design a region design does it a disservice. It creates confusion that could be avoided. I thought I caught the gist of what was going on from the title and the short description at the top. Only in the detailed design doc did I realize how wrong my assumptions had been.

This design is not doing region management where
a block of memory does not need to be Garbage collected (swept) and the
entire block of memory is released at once. Instead the GC still runs over these
blocks, the blocks are swept for free space upon allocation, and the GC is
run eagerly when the user annotation point is returned from.

So I would, foremost, suggest a rename.

Any of these names would be better...

immix.Do()
blue.Do()
unshared.Do()
collectimmediately.Do()
eagersweep.Do()
...
your idea here
...

This would also immediately allow the detailed design doc to distinguish the
new implicit approach by name. This would help the narrative in many places where this
design and an arena/region approach are to be contrasted.

[timothy-king]

Would runtime diagnostics for fades be feasible? Particularly reporting the location of the pointer write that causes the fade.

I could imagine some users of regions wanting fairly rigorous enforcement of not accidentally copying memory allocated within a region. Runtime diagnostics seem like a plausible way to deliver on this.

[mknyszek]

Yup! There's a section on it in the doc. Reporting the location of the pointer write that caused the fade is the "Fade profile" sub-heading. This would be fairly easy to implement and could easily be sampled, so we can control overheads.

[kelindar]

I read the detailed proposal, and the papers its inspired by. Overall, I can see myself using this in a few places which would bring some sanity back to the codebase.

I think shipping good tooling together with this proposal would actually be key to making it much more useful. If we have something like the proposed heap with lifetime analysis, it would be much easier to identify good candidates. Personally, pprof would be preferred approach since we have it live in production and already use it to identify bottlenecks.

[tandr]

Do I understand proposed spec properly - region exists only for the enclosed function, and not functions called or goroutines created from it? If so, why region is not "inherited"?

[mknyszek]

@CannibalVox It depends. If they're making RPCs and deserializing, for example, JSON, then only passing on a handful of fields out of a larger value (by "weight," so it's really the byte count that matters) it may make a substantial difference. Constructing small regions for these sorts of things is much more feasible than small arenas (in the arenas experiment specifically) because the up-front memory costs are much lower.

Though I'm not sure it's super useful to talk about such situations in the abstract. I do that a little bit above and in the design doc, but I try to keep it motivated by real examples (if not explicitly stated, for various reasons, then at least in principle). It would be helpful in general to get feedback on whether developers believe they could make relatively easy and effective use of regions in the applications they're actually working on.

I've gotten some of this feedback already, and I've received both positive and negative responses (gopls, for example, launches goroutines to do sub-tasks, and actually keeps around all the memory each goroutine produces until the overall task is complete). I do think there's a good amount of upside here, despite the situations where regions don't really apply. And it's OK, IMO, if they don't apply absolutely everywhere.

[robaho]

Isn't the json use case already covered using pooling of decoding buffers? I suspect that is the largest dynamic memory usage. I also suspect that decoding out of the buffers should almost always be stack allocations.

The more complex fan-out to multiple services and aggregate seems a more natural and useful fit for regions. If regions are going to work across Go routines, then aren't the existing allocation profilers sufficient to allow developers to restructure their code to perform more stack/pooled allocation rather than dynamic heap? (although as I stated from the beginning, the less manual memory management and tuning the better imo).

If the usage is isolated to a function it seems the arena experiment and the explicitness is all that is required (although I dislike the lack of safety).

[mknyszek]

Isn't the json use case already covered using pooling of decoding buffers? I suspect that is the largest dynamic memory usage. I also suspect that decoding out of the buffers should almost always be stack allocations.

The decoding buffers are just one piece of it. The deserialized forms tend to result in many allocations. Many are wasted, especially if only a small part of a response is used. This is true of other serialization libraries too, like protobuf. There are ways to work around this at the library level, but there's only so much you can do. It's also just one example, this design is not targeting solely RPC services.

If regions are going to work across Go routines

I do not think they will. See my response here.

If the usage is isolated to a function it seems the arena experiment and the explicitness is all that is required

Frequently, it is not isolated to a specific function. The virality caused due to the explicitness is a big reason why arenas are on indefinite hold (I went into this in detail in the "Background" section above). That's also why region.Do applies to f and its callees, to bring this point back to the overall thread topic.

[robaho]

Thanks. I didn't realize that a region working across Go routines was out of scope/impossible with the design.

With that, I guess I will assert my original opinion and drop out. It seems like a complication to the language and readability that I doubt will bring enough bang for buck and I think that doing something transparently behind the scenes with GC efficiency would be more worthwhile and more Go-like. (I remember when Java did "real-time Java" RTSJ and it was slog - mainly because it was writing C in Java - it eventually became a commercial offering, https://en.wikipedia.org/wiki/Real-time_Java - this is far smaller in scope but I see similar issues in the manual memory management).

I fully appreciate the work you put in and the cleverness of the design! Well done.

[CannibalVox]

it's really the byte count that matters

This is pretty unusual when talking about go allocations, isn't it? I know that one of the little "tricks of the trade" some use with go is to allocate a slice instead of multiple individual objects because most of the allocation costs are per-allocation rather than per-byte. Are the per-byte costs of a fade in addition to the per-allocation costs we usually pay for a heap allocation, and if so, the cost of several small fades is still pretty substantial.

[RLH]

I’m sympathetic to the “I don’t want to think like a garbage collector” argument but not so much to the "I don't want to think about control flow and object availability” argument. If done right the core infrastructure needed to support regions and fading builds on infrastructure and tools to better understand and manage object availability. The proposal lists a few such tools but doesn’t articulate the overall goal of such tools which goes well beyond servicing memory management to exploring and debugging object availability. A separate proposal to add these tools would be a great addition to the Go ecosystem.

Common Lisp and Guy Steele created some terminology that can be used in such a proposal. See 3. Scope and Extent. Briefly objects are divided based on their availability. Availability determined the lexical scope has lexical extent. Availability that is bounded by dynamic scope has dynamic extent and globally available objects have indefinite extent. With these three concepts developers can reason about their object’s availability without having to think like a garbage collector. Reasoning about control flow and data availability is at the heart of concurrent programming and debugging such programs.

The problem being worked on here is how to more efficiently manage memory for objects with dynamic extent. Currently the GC treats objects with dynamic extent and indefinite extent the same. This proposal says we can do dynamic extent better. Since this proposal is focused on the GC aspects of the discussion it sometimes conflates the problem with implementation terminology which is understandable. A separate proposal talking about availability, independent of memory management, would use terms like “extent” and “scope”. A proposal about memory management would build on the "extent" proposal and use region, fade, bound and so forth. More importantly the “extent” tools proposal could be accepted even if the “region” proposal is declined for performance reasons.

[RLH]

Yes. Objects with dynamic extent are implemented with regions. When an object with dynamic extent is published and becomes one with indefinite extent implementation uses unbinding. As much as possible code should be architected to have lexical or dynamic extent instead of indefinite extent. The resulting code is less fragile since the reasoning is more localized. For example objects with dynamic extent don't need finalizers or weak references since the code knows where the object's extent ends. We have the tools to reason about global variables and lexical scope but not about an object's extent. Let's change that.

[robaho]

As an aside, @RLH that seems like a step backwards... the benefit for GC based languages is not having to think about extent (if I am understanding how you're using the term extent). As an example, if GC collections was infinitely efficient (e.g. ultra cheap write barriers in hardware - like the Azul chips), there would be no need for regions. A region in this case is only being created to aid the garbage collector in being more efficient.

[RLH]

Extents exist today and all that is being proposing is that we articulate what extents are and provide tools to reason about them. In the future you wouldn't need to use these tools or even think about extents. If you treat an object with indefinite extent like it has dynamic extent it will have exactly the same bugs it has today. The tools will simple help to find the bug.

While Go doesn't have a name for it yet, the only place Go provides tools for reasoning about extents is to reason about lexical extent in service of improving escape analysis. Some of the tools for dynamic extent may need code to mark an interesting dynamic scope, such as region.Do, but semantics, safety, and correctness will not be changed.

[robaho]

@RLH I read the Lisp extent reference. All regular objects in Go have indefinite extent. It is based on reachability. The fact that an object may be allocated on the stack for more efficient garbage collection (or lack there of) is immaterial - that is simply an optimization. The same holds for an object allocated in a region in this proposal - they are indefinite extent. (Contrast, the area experiment was similar to unsafe memory allocation - these are not indefinite extent).

I hope Go doesn't ever allow anything other than indefinite extent...

[RLH]

Not sure how to make progress here. Perhaps I misremembered the Common Lisp spec or I'm misusing the terms. In any case my definition is an object allocated in a lexical scope that is not available outside that scope has lexical extent. Such objects clearly have properties that all objects don't have. Since the are not available to other goroutines they have no race hazards. Since the object is not available to called procedures they don't have to be reloaded upon return, in general they can be reasoned about locally. They also have properties invisible to the program such as being stack allocated. These are not immaterial properties and programmers take advantage of these properties all the time. This is all allowed today, it's just an observation.

[dominikh]

@mknyszek regarding Diagnostics, would it be feasible to have a new kind of runtime/trace span corresponding to memory regions and to record the number/size of allocations as well as the number/size of fades?

[mknyszek]

Yup, we could have a runtime/trace.Region for a region.Do created automatically.

We could also insert profile samples from some of the proposed profiling diagnostics into the trace, like we do for CPU profiling. Recording the approximate size and number of allocations would be OK, but anything more granular would probably be too expensive as a default. (If we have an opt-in option for expensive tracing events in the future, we could do that. For now, that would fall under an "experimental" tracing event, which might be good enough for some use-cases.)

[kurahaupo]

I wouldn't rule out rebinding from an inner to an outer region, at least in the spec, but I would make it optional so that the initial implementation doesn't have to do so.

This is kinda like how resizing a slice may - or may not - relocate the unchanged part, and code using such slices is expected to cope with both possibilities.

[mknyszek]

For sure, my intent was not to rule it out.

[tandr]

Sorry, one more (naive) question.
If I understood spec correctly, GC is pretty much disabled for this region of memory, memory is pinned, and still-referenced things will be copied out on exit from goroutine. Would that mechanism be extended to handle interop with non-gc languages? Say, in case of C, could whole region be malloc-ed , whatever happens inside of C func happens, and on exit from function referenced results would be automatically copied out to GC-controlled region?

Sorry if this is way too far for the current spec.

[mknyszek]

There's no copying involved. See the detailed design doc. It's fine to pass this memory to C (possibly without fading, though explicit pinning with runtime.Pinner will probably have to force a fade) but it's an open question as to what to do if that C code calls back into Go again.

[CAFxX]

While manual region.Do annotations provide good control, I think that a (possibly opt-in) sampling-based runtime mechanism would be more ergonomic and potentially more effective. By leveraging existing profiling infrastructure, such an approach would allow the runtime to dynamically adapt region allocation to almost all workloads, optimizing performance transparently and without requiring explicit developer intervention. This design would align more closely with Go's philosophy of automatic memory management and would reduce the burden on developers. Similar to the decision to avoid an inline hint due to potential misuse, a runtime-managed region system mitigates similar risks by automating decisions based on actual profiling data, while avoiding the problems highlighted in the request-oriented collector experiment.

[atdiar]

Yes but that's virtually almost everywhere?

What's the tradeoff then, binary size? Branch misprediction perhaps?

Not saying it's not possible. Seems a bit weird.

[CAFxX]

Yes but that's virtually almost everywhere?

Yes. FWIW we do this already for stack size check (at ~every function entry) where you see a variation of the following (on x64):

CMPQ SP, 0x10(R14)
JBE <address>

where <address> is the PC of a small fragment of code, present at the end of each function, where the call to resize the stack is executed before jumping back to the start of the function, e.g.:

CALL runtime.morestack_noctxt.abi0(SB)
JMP <start of function>

FWIW it may be very well possible to "fold" the portal into the stack check machinery, thereby not adding any overhead in the disabled case. I did something similar when I implemented the dynamic initial goroutine stack size estimation experiment.

What's the tradeoff then, binary size? Branch misprediction perhaps?

Binary size I'd guess very low single digit %. Branch misprediction should not be a problem in a ideal world (because these branches would be extremely predictable), but in practice in some cases any additional branch can alias with others in the predictor (because the predictors have finite sizes).

So I would argue these are not the main issues. I suspect people will mostly initially balk at the lack of determinism/predictability (even though the whole GC aspect of the runtime would then need to be painted in the same light). It is worth pointing out that in theory one could make this static after a profiling run, or disable it altogether, thereby achieving determinism.

The other issue is that it may take a bit (~few seconds?) after startup for the "optimal" regions to be identified and exploited ("warmup"). In this transient the performance characteristics of the application would differ from the ones after the warmup has completed (but as previously mentioned, a profiling run could solve this).

[atdiar]

Apart from the fact (a bit tongue-in-cheek) that it will require very good profiling ux (since it seems people can detect supply chain attacks by perf monitoring, although they should optimally be mitigated upstream), I guess that could be explored.

[szabba]

The other issue is that it may take a bit (~few seconds?) after startup for the "optimal" regions to be identified and exploited ("warmup"). In this transient the performance characteristics of the application would differ from the ones after the warmup has completed (but as previously mentioned, a profiling run could solve this).

I think the time this takes matters a lot. If it really is a couple of seconds, no one will complain.

I've seen Java applications with peak ~100 rps per application instance that had lowered performance for 2 days after deployment. Obviously there'd be many technical differences between what's outlined in this thread and a JVM JIT. But they're both runtime optimizations driven by sampled-at-runtime data. How likely is it this would lead to similar... performance shenanigans?

[mknyszek]

At the end of the portal's evaluation mode execution, a dummy reclaim operation would be performed to determine whether the allocations made during that execution have outlived it

I think the part I bolded here is hiding a lot of complexity. I don't think that it's actually possible to track the liveness as it pertains to all possible 'portals' on the current goroutine through just the write barrier cheaply, or anywhere nearly as cheaply as the write barrier described in the detailed design doc. (Unless of course I'm misunderstanding.)

EDIT: I wrote a lot more but I don't think it was very coherent.

(We have considered ways of determining how far up-stack a value escapes using a combination of static and dynamic information (so, PGO), at which point we could just stack allocate the value in some caller. One could even just stack allocate it optimistically. The difficulty was always figuring out how far up-stack something goes in the first place.)

[atdiar]

Just wondering: if we were able to write generic code working for all functions
(cf. issue on variadic generics), region.Do could accept any function.

Then we could perhaps enforce a region to not rely on implicit global state but having state passed as argument.
Also, the only fades would be the return values and anything that gets stuck to the input arguments by assignment (it's transitive via pointers).

Edit: and objects which assign pointers created in the region to implicitly available global objects (via methods for instance). It's still interprocedural.

Perhaps that it could then become statically sound and (more) precise.

Just thinking out loud, I might be forgetting some important details.

[edit2] thinking about this again, not sure it brings much more.
If tooling can find out the scope in which a variable is declared, then the point is moot.
Can ignore.

[szabba]

I don't see how this would fit together. Go functions can close over both global and local state whether they accept arguments or not. They can prolong the lifetime of a value allocated in the body whether they return a value or not.

I suppose that a function passed region.Do does not do that could be enforced with runtime support and panic on violation... but then the function passed to region.Do is on the hook for when any functions it calls does that. If those are 3rd party library functions and the behavior wrt this changes in a transitive dependency after an update... that'll be a fun surprise. This would be a poorly composable footgun. I don't think it's compatible with the proposal goals.

Am I misunderstanding what you're imagining?

[robaho]

I don't understand why the PGO "solution" isn't sufficient. The runtime tracks the total number of objects and their sizes allocated from each call point and tracks where each object is allocated. Then at a GC cycle, it determines the number of objects freed that were allocated from each call point down, against the number of objects still live from that call point. The functions with the highest free/live ratio are good candidates for an implicit region. You could also exclude regions to only those call sites where the live ~= 0.

Since the regions are "safe", even if it makes a mistake it should be no worse than the worst case of a developer adding the hint at the wrong place.

If the claim is that a developer doing this manually will be 100% correct, even with library usage, then you truly are moving Go into a manual memory management scenario.

[atdiar]

I don't see how the runtime could figure that out without static hints.

This would be very non-deterministic. In a way that even GC isn't.

[robaho]

Not the runtime, PGO. I described the process in the post. I think all of these are already captured in some form, in order to allow pprof memory profiling to work, it would probably need some changes through, but doing this via PGO vs programmers manually managing the regions seems a better solution than what is proposed.

[mknyszek]

I don't understand why the PGO "solution" isn't sufficient. The runtime tracks the total number of objects and their sizes allocated from each call point and tracks where each object is allocated. Then at a GC cycle, it determines the number of objects freed that were allocated from each call point down, against the number of objects still live from that call point. The functions with the highest free/live ratio are good candidates for an implicit region. You could also exclude regions to only those call sites where the live ~= 0.

As I've mentioned elsewhere, I think the fundamental problem with this approach is the heuristic. What is a good heuristic that also doesn't require data that is expensive to collect? I don't think we have a good answer to that question.

For example the one you describe here, I suspect, would not work very well in practice. Using the GC as a signal for lifetime tracking is a poor signal for regions because from the GC's perspective, lifetimes are measured in whole GC cycles. With regions, you're in the best possible place if you're reusing region memory many times in a single GC cycle. The timescales are different enough that I suspect there's not much of a connection between whether an object does not survive even a single GC cycle and whether it would make sense to put it in a region. For a real-world example, I suspect that gopls would likely get slower with PGO if we applied this heuristic.

(That being said, with a prototype, this wouldn't be all that hard to try, and it would be possible to explore other heuristics too. I'm just personally somewhat pessimistic that we'll find something good here.)

Frankly, I sympathize with your stance. It would be better if regions could be applied automatically and that just made your code go faster. I think the most promising step forward in that vein is the opt-in-request-oriented-collector-with-PGO-disabling-per-goroutine approach described at the top, and it's something that, if we get to a prototype, I definitely think is worth exploring. It is annoying to change your code to work with regions, especially if you feel like your case is really easy (request-oriented, heh). It's much easier to just set a flag and try things out, possibly integrated with your PGO pipeline. And, I think it's a much easier problem to find where regions aren't working than it is to find where they might work, especially if it's only ever at the goroutine level.

(That being said, I still think the API is OK, and I am aware of at least some projects which would benefit from the additional control.)

[robaho]

What is a good heuristic that also doesn't require data that is expensive to collect?

I guess that is what I don't understand. Why does it matter if it is expensive for PGO to collect? When you run under PGO you expect the program to run much slower, as it is recording aspects to make a future compilation faster. So even if it is expensive, who cares, especially if it automates the region specifications.

I am not trying to diminish what you've done - I think it is great. I am just afraid that is forges a path forward for Go that requires "manual" tuning - which I believe to be the bane of most systems - especially complex/enterprise ones. If people think they can do that better than the compiler/language authors, use C++ - if not, I think the optimizations should come from restricting the language in order to allow more compiler optimizations - otherwise things become a real mess.

[mknyszek]

When you run under PGO you expect the program to run much slower

I think this is incorrect. The overhead today is a couple percent at most.

What I mean by "expensive" is 2x or 10x or 100x. This is a real hindrance for two important reasons:

1. The slower your code runs under profiling, the less it reflects reality. (A CPU profile of a heavily-instrumented and slowed-down program is not going to accurately represent how that program runs without instrumentation, which affects the quality of the data collected.)
2. It reduces the chance that someone will actually want to run this kind of profiling in production, since it will have to affect real traffic. This lowers the utility of PGO.

The slowdown for the race detector, for example, is about 2x, and to my knowledge it is not used widely outside of testing as a result. I don't think it's very useful to profile race-instrumented code either, due to (1). (There were other limitations that made it hard to use in production in the past, such as a limit on the number of goroutines, and some of those have been resolved. It's unclear to me how that has changed things.)

I am not trying to say that there isn't a way to collect the necessary information efficiently, only that this is an open-ended research problem with, at the moment, no clear path forward and likely a long time horizon.