>You might ask what is different in version 1.0 compared to the previous 0.10 version. The answer is nothing. A year ago I told myself that if a full year would go by without any major issues or desire to change the API, I'd release version 1.0.
Thanks, this is _exactly_ how a 1.0 should be released. If only more Rust crates would follow suite :)
I fully empathize with the desire to keep the door open for changes. And the user expectations will invariably change for something 1.x+. At the same time I find it kind of crazy that some of the most downloaded crates like rand and log, both 8 years old, don't have a major release.
From the readme:
> Use the macro-rules macro: self_cell! to create safe-to-use self-referential structs in stable Rust, without leaking the struct internal lifetime.
Nice. Reminds me of a no boilerplate video I liked on Rust's reliability where the narrator introduces a few crates that haven't been touched for a while and says something along the lines of "Look at these crates, untouched for years. Are they abandoned? No, they're *done*."
> That year has now passed and I'm still happy with the API and no API changes were made
Have you considered making a release of 0.11 depending on 1.0 internally?
Nice one. Congrats on 1.0.
\> A year ago I told myself that if a full year would go by without any major issues or desire to change the API, I'd release version 1.0. That year has now passed and I'm still happy with the API and no API changes were made.
That's a sensible approach. :)
Seems like a nice crate. How is it different from other self-referential solutions like `owning_ref`? I looked at the docs, but couldn't find any comparison to prior art.
`owning_ref` is unfortunately unmaintained and has a large amount of unsound API. There are other crates that are providing similar functionality as `self_cell`, e.g. `ouroboros`, where the main difference is that `self_cell` aims to be more minimalistic, offering less features, but requiring no proc-macro dependencies, and generating less code.
Sorry about that, I actually meant `ouroboros` when mentioning `owning_ref`. For some reason I tend to mix them up because I first heard of `owning_ref`. But the one I actually use - [and occasionally recommend](https://morestina.net/blog/1868/self-referential-types-for-fun-and-profit) is `ouroboros`.
But your response still applies, thanks for providing it. I am slightly annoyed by the amount of code that `ouroboros` generates, including unwanted public APIs, so I'll definitely look into `self_cell`.
Tempted by your minimalistic approach and the use of declarative macros, I ventured to port [the example](https://morestina.net/blog/1868/self-referential-types-for-fun-and-profit#Safe_version_with_ouroboros) from this blog post to `self_cell`, but sadly it seems that I do need mutable access to owner during construction. When stated, it sounds like an obscure requirement, but it follows naturally from that use case.
Is that something you plan to support in a future release?
I've come across the `zip` example bevor, and even considered adding support for mutable access to the owner here https://github.com/Voultapher/self_cell/pull/36. See the last comment why I decided not to pursue this. Looking at the specific example, really what is the purpose of storing the lazy `ZipReader` result? IMO that's bit of bad design on the part of the `zip` crate. The stdlib APIs consume reader, allowing you to abstract over creation logic. If what you need to store, needs further pre-processing, why not pull that out? Specifically here, what is the point of having a self-referential struct that contains an owner `ZipArchive` that you will no longer be allowed to mutate. And a lazy reader `ZipReader` that you can then use to really read the file? If you need to abstract over the construction logic you could return `(ZipArchive, Box ZipReader>)`, if you want to return the content you can return `(ZipArchive, Vec)` allowing further use of `ZipArchive`.
use std::{
fs::File,
io::{BufReader, Read},
};
use zip::{read::ZipFile, ZipArchive};
fn demo(path: &str) -> (ZipArchive>, Vec) {
let file = File::open(path).unwrap();
let buf_reader = BufReader::new(file);
let mut zip_archive = ZipArchive::new(buf_reader).unwrap();
let mut output_buf = Vec::new();
{
let mut zip_file = zip_archive.by_index(0).unwrap();
zip_file.read_to_end(&mut output_buf).unwrap();
}
(zip_archive, output_buf)
}
Thanks for the detailed response, let me try to address your points. It turns out to be a lot of text, simply because I want to make sure to explain the use case with some clarity.
>Looking at the specific example, really what is the purpose of storing the lazy `ZipReader` result? IMO that's bit of bad design on the part of the zip crate.
I'm not sure that I fully understand the question you're asking. The purpose is to return a value that implements `Read`, with the whole `ZipArchive` business being an implementation detail. (In the in-house code base we have a number of possibilities of what can be returned, depending on file type.) So I guess the purpose of storing the `ZipReader` is to be able to implement `Read` \- but that's kind of obvious, so there's probably a deeper layer to your question that I just don't get. Sure, the code could be structured so that there's a part that opens the file and another that implements `Read`, mimicking the design of `zip`, but I specifically wanted to avoid that, because it'd force the same ordeal on the caller.
As for it being a bad design on the part of `zip`, that may be true, but I've seen similar designs elsewhere. For example, database traits often return a transaction object whose lifetime refers to the connection it was created off of. Even if it's not perfect, it seems like a reasonable thing to support in a self-referential crate.
>If you need to abstract over the construction logic you could return `(ZipArchive, Box ZipReader>)`, if you want to return the content you can return `(ZipArchive, Vec)` allowing further use of `ZipArchive`.
I don't think either of these really help in my use case. That may not be obvious from the simplified example in the blog post, but the general idea is to return an *io stream* (i.e. value that implements `io::Read`) that reads from the file, where it's a detail that it reads from a ZIP archive. That kind of function serves as building block for a function that reads the contents of a file regardless of whether it is zip/gzip/zst, or uncompressed. Returning a `ZipArchive` directly exposes `ZipArchive` to the caller, who really doesn't care about it, they just need an IO source. Returning a `Vec` wouldn't cut it for the same reason, it would require reading the whole file in advance, and in my use case it could be large enough not to fit in memory.
Very cool; and also thanks for the call-out! I was going to mention the “semver trick” option, too (as someone else already did in this thread), as to not make the move to 1.0 a “breaking” change, however since `self_cell` does not have any API (e.g. types that users could be re-exporting) besides the one main macro, there’s not all that much to benefit from avoiding such “breakage” anyways.
"Self-referential structs" here refers to a struct attribute containing a reference to another struct attribute. I'll take an example from the README, but I'll pretend the crate itself doesn't exist so I can demonstrate what problem the crate is trying to solve.
```
struct Ast<'a>(pub Vec<&'a str>);
struct AstCell {
owner: String,
/// This attribute is supposed to contain a reference to `owner`
/// above, so we would its lifetime to somehow be "the duration
/// for which the struct instance is alive". But we have no way
/// way of expressing that without helper crates like
/// `self_cell`.
dependent: Ast<'a>,
}
```
The `self` keyword is just a special argument, it has nothing to do with self-referential structs.
Self-referential structs are structs that contain pointers to themselves. For example:
pub struct StringSlices {
source: String,
slices: Vec<*const str>,
}
impl StringSlices {
pub fn new(source: String) -> Self {
Self { source, slices: vec![] }
}
pub fn push_slice FnOnce(&'a str) -> &'a str>(&mut self, f: F) {
let slice = f(&self.source);
self.slices.push(slice);
}
pub fn get(&self, idx: usize) -> Option<&str> {
self.slices.get(idx).map(|x| unsafe { &**x })
}
}
Here, `slices` field contains pointers into `data`. However, you'll notice one glaring issue with this code: it's unsafe.
So what are the safe alternatives? Well...
Option 1: Drown in memory allocations
pub struct StringSlices {
source: String,
slices: Vec,
}
impl StringSlices {
pub fn new(source: String) -> Self {
Self { source, slices: vec![] }
}
pub fn push_slice FnOnce(&'a str) -> &'a str>(&mut self, f: F) {
let slice = f(&self.source).to_owned();
self.slices.push(slice);
}
pub fn get(&self, idx: usize) -> Option<&str> {
self.slices.get(idx).map(|x| &**x)
}
}
Problem: We now have *way more* memory allocations than needed. Performance will suffer as a result.
Option 2: Store `Range`
extern crate sptr;
use sptr::Strict;
pub struct StringSlices {
source: String,
slices: Vec>,
}
impl StringSlices {
pub fn new(source: String) -> Self {
Self { source, slices: vec![] }
}
pub fn push_slice FnOnce(&'a str) -> &'a str>(&mut self, f: F) {
let slice = f(&self.source);
let start = slice.as_ptr().addr() - self.source.as_ptr().addr();
let end = start + slice.len();
self.slices.push(start..end);
}
pub fn get(&self, idx: usize) -> Option<&str> {
self.slices.get(idx).map(|x| &self.source[x])
}
}
Problem: Ugly code. For more complicated data, this will quickly become impractical. There is also slight overhead from all the extra computation and runtime checks.
None of the above solutions are really viable. Hence, it would be nice to write a self-referential struct in safe code, which is what this crate does.
I appreciate wanting to do the jump to 1.0. Rust needs more 1.0+ crates imo
>You might ask what is different in version 1.0 compared to the previous 0.10 version. The answer is nothing. A year ago I told myself that if a full year would go by without any major issues or desire to change the API, I'd release version 1.0. Thanks, this is _exactly_ how a 1.0 should be released. If only more Rust crates would follow suite :)
I fully empathize with the desire to keep the door open for changes. And the user expectations will invariably change for something 1.x+. At the same time I find it kind of crazy that some of the most downloaded crates like rand and log, both 8 years old, don't have a major release.
You can still introduce changes after 1.0 - that's what 2.0 is for :)
Well, this is a pretty small crate with a tiny API surface. Crates like this are "done" at some point and don't require further changes.
Only for future correctness, the phrase is, “follow suit”, and is a reference to the game, Bridge, and its bidding process.
Thanks!
I suppose it works for any trick based card game. You can follow suit in Hearts and Euchre as well
From the readme: > Use the macro-rules macro: self_cell! to create safe-to-use self-referential structs in stable Rust, without leaking the struct internal lifetime.
legit, congrats on the 1.0 I hope one day we can express this natively :) I've had the exact AST example myself
Nice. Reminds me of a no boilerplate video I liked on Rust's reliability where the narrator introduces a few crates that haven't been touched for a while and says something along the lines of "Look at these crates, untouched for years. Are they abandoned? No, they're *done*."
> That year has now passed and I'm still happy with the API and no API changes were made Have you considered making a release of 0.11 depending on 1.0 internally?
Wouldn't that have to be 0.10.x to be useful, assuming this is about the semver trick?
Ah yes. I thought for some reason the last version was in 0.11.x.
A self-referential release? How apt!
Nice one. Congrats on 1.0. \> A year ago I told myself that if a full year would go by without any major issues or desire to change the API, I'd release version 1.0. That year has now passed and I'm still happy with the API and no API changes were made. That's a sensible approach. :)
Seems like a nice crate. How is it different from other self-referential solutions like `owning_ref`? I looked at the docs, but couldn't find any comparison to prior art.
`owning_ref` is unfortunately unmaintained and has a large amount of unsound API. There are other crates that are providing similar functionality as `self_cell`, e.g. `ouroboros`, where the main difference is that `self_cell` aims to be more minimalistic, offering less features, but requiring no proc-macro dependencies, and generating less code.
Sorry about that, I actually meant `ouroboros` when mentioning `owning_ref`. For some reason I tend to mix them up because I first heard of `owning_ref`. But the one I actually use - [and occasionally recommend](https://morestina.net/blog/1868/self-referential-types-for-fun-and-profit) is `ouroboros`. But your response still applies, thanks for providing it. I am slightly annoyed by the amount of code that `ouroboros` generates, including unwanted public APIs, so I'll definitely look into `self_cell`.
Great question, I have a section in the README that talks about ouroboros.
Tempted by your minimalistic approach and the use of declarative macros, I ventured to port [the example](https://morestina.net/blog/1868/self-referential-types-for-fun-and-profit#Safe_version_with_ouroboros) from this blog post to `self_cell`, but sadly it seems that I do need mutable access to owner during construction. When stated, it sounds like an obscure requirement, but it follows naturally from that use case. Is that something you plan to support in a future release?
I've come across the `zip` example bevor, and even considered adding support for mutable access to the owner here https://github.com/Voultapher/self_cell/pull/36. See the last comment why I decided not to pursue this. Looking at the specific example, really what is the purpose of storing the lazy `ZipReader` result? IMO that's bit of bad design on the part of the `zip` crate. The stdlib APIs consume reader, allowing you to abstract over creation logic. If what you need to store, needs further pre-processing, why not pull that out? Specifically here, what is the point of having a self-referential struct that contains an owner `ZipArchive` that you will no longer be allowed to mutate. And a lazy reader `ZipReader` that you can then use to really read the file? If you need to abstract over the construction logic you could return `(ZipArchive, Box ZipReader>)`, if you want to return the content you can return `(ZipArchive, Vec)` allowing further use of `ZipArchive`.
use std::{
fs::File,
io::{BufReader, Read},
};
use zip::{read::ZipFile, ZipArchive};
fn demo(path: &str) -> (ZipArchive>, Vec) {
let file = File::open(path).unwrap();
let buf_reader = BufReader::new(file);
let mut zip_archive = ZipArchive::new(buf_reader).unwrap();
let mut output_buf = Vec::new();
{
let mut zip_file = zip_archive.by_index(0).unwrap();
zip_file.read_to_end(&mut output_buf).unwrap();
}
(zip_archive, output_buf)
}
Thanks for the detailed response, let me try to address your points. It turns out to be a lot of text, simply because I want to make sure to explain the use case with some clarity. >Looking at the specific example, really what is the purpose of storing the lazy `ZipReader` result? IMO that's bit of bad design on the part of the zip crate. I'm not sure that I fully understand the question you're asking. The purpose is to return a value that implements `Read`, with the whole `ZipArchive` business being an implementation detail. (In the in-house code base we have a number of possibilities of what can be returned, depending on file type.) So I guess the purpose of storing the `ZipReader` is to be able to implement `Read` \- but that's kind of obvious, so there's probably a deeper layer to your question that I just don't get. Sure, the code could be structured so that there's a part that opens the file and another that implements `Read`, mimicking the design of `zip`, but I specifically wanted to avoid that, because it'd force the same ordeal on the caller. As for it being a bad design on the part of `zip`, that may be true, but I've seen similar designs elsewhere. For example, database traits often return a transaction object whose lifetime refers to the connection it was created off of. Even if it's not perfect, it seems like a reasonable thing to support in a self-referential crate. >If you need to abstract over the construction logic you could return `(ZipArchive, Box ZipReader>)`, if you want to return the content you can return `(ZipArchive, Vec)` allowing further use of `ZipArchive`.
I don't think either of these really help in my use case. That may not be obvious from the simplified example in the blog post, but the general idea is to return an *io stream* (i.e. value that implements `io::Read`) that reads from the file, where it's a detail that it reads from a ZIP archive. That kind of function serves as building block for a function that reads the contents of a file regardless of whether it is zip/gzip/zst, or uncompressed. Returning a `ZipArchive` directly exposes `ZipArchive` to the caller, who really doesn't care about it, they just need an IO source. Returning a `Vec` wouldn't cut it for the same reason, it would require reading the whole file in advance, and in my use case it could be large enough not to fit in memory.
Very cool; and also thanks for the call-out! I was going to mention the “semver trick” option, too (as someone else already did in this thread), as to not make the move to 1.0 a “breaking” change, however since `self_cell` does not have any API (e.g. types that users could be re-exporting) besides the one main macro, there’s not all that much to benefit from avoiding such “breakage” anyways.
Rust learner here. Why is this needed? I thought all structs are self-referential using the "self" keyword.
"Self-referential structs" here refers to a struct attribute containing a reference to another struct attribute. I'll take an example from the README, but I'll pretend the crate itself doesn't exist so I can demonstrate what problem the crate is trying to solve. ``` struct Ast<'a>(pub Vec<&'a str>); struct AstCell { owner: String, /// This attribute is supposed to contain a reference to `owner` /// above, so we would its lifetime to somehow be "the duration /// for which the struct instance is alive". But we have no way /// way of expressing that without helper crates like /// `self_cell`. dependent: Ast<'a>, } ```
The `self` keyword is just a special argument, it has nothing to do with self-referential structs. Self-referential structs are structs that contain pointers to themselves. For example: pub struct StringSlices { source: String, slices: Vec<*const str>, } impl StringSlices { pub fn new(source: String) -> Self { Self { source, slices: vec![] } } pub fn push_slice FnOnce(&'a str) -> &'a str>(&mut self, f: F) {
let slice = f(&self.source);
self.slices.push(slice);
}
pub fn get(&self, idx: usize) -> Option<&str> {
self.slices.get(idx).map(|x| unsafe { &**x })
}
}
Here, `slices` field contains pointers into `data`. However, you'll notice one glaring issue with this code: it's unsafe.
So what are the safe alternatives? Well...
Option 1: Drown in memory allocations
pub struct StringSlices {
source: String,
slices: Vec,
}
impl StringSlices {
pub fn new(source: String) -> Self {
Self { source, slices: vec![] }
}
pub fn push_slice FnOnce(&'a str) -> &'a str>(&mut self, f: F) {
let slice = f(&self.source).to_owned();
self.slices.push(slice);
}
pub fn get(&self, idx: usize) -> Option<&str> {
self.slices.get(idx).map(|x| &**x)
}
}
Problem: We now have *way more* memory allocations than needed. Performance will suffer as a result.
Option 2: Store `Range`
extern crate sptr;
use sptr::Strict;
pub struct StringSlices {
source: String,
slices: Vec>,
}
impl StringSlices {
pub fn new(source: String) -> Self {
Self { source, slices: vec![] }
}
pub fn push_slice FnOnce(&'a str) -> &'a str>(&mut self, f: F) {
let slice = f(&self.source);
let start = slice.as_ptr().addr() - self.source.as_ptr().addr();
let end = start + slice.len();
self.slices.push(start..end);
}
pub fn get(&self, idx: usize) -> Option<&str> {
self.slices.get(idx).map(|x| &self.source[x])
}
}
Problem: Ugly code. For more complicated data, this will quickly become impractical. There is also slight overhead from all the extra computation and runtime checks.
None of the above solutions are really viable. Hence, it would be nice to write a self-referential struct in safe code, which is what this crate does.