https://blog.ipfs.io/ https://blog.ipfs.io/img/ipfs-logo-256-ice.png https://blog.ipfs.io/ Recent content on Hugo -- gohugo.io en Thu, 11 Jun 2020 00:00:00 +0000 Decentralized Identity, IPFS and ION https://blog.ipfs.io/2020-06-11-identity-ipfs-ion/ Thu, 11 Jun 2020 00:00:00 +0000 https://blog.ipfs.io/2020-06-11-identity-ipfs-ion/ Identity is a key part of how we all interact with each other on the internet every day—sometimes every moment of every day. IPFS logo and ION logo

Identity is a key part of how we all interact with each other on the internet every day—sometimes every moment of every day. And sometimes each raging moment as we do the “forgot your password?” dance yet again. Regardless, forging a path forward for user-controlled online identity is a table-stakes requirement for the dweb to succeed.

But, identity is hard. There are no easy solutions for interoperable decentralized digital assertions of self. Privacy, security, validity, access control, surveillance, GDPR, KYC, anonymity. It’s like Inception but each nested dream level is the same minefield, and those minefields are littered with projects that’ve tilted at the identity windmill, leaving naught behind but vinyl stickers slowly being covered up on our laptops.

Hope has not forsaken this land. The Decentralized Identity Foundation has been plugging away at this challenge for many years, as have groups like Rebooting the Web of Trust. And there’ve been efforts to build decentralized identity systems on top of IPFS, such as IPID, Nomios.io, the IPFS IDM, and most recently the Ceramic Network.

However, today we’re celebrating the launch of a decentralized identity protocol and service from an unlikely place: Microsoft.

Microsoft has been increasingly present in open-source tools and services in the past few years, and has now launched a standards-based decentralized identity service called ION.

ION logo

ION has been under development for over a year, and is an instance implementation of Sidetree (a blockchain-agnostic distributed PKI protocol) that runs on the Bitcoin blockchain.

And it stores transaction data on IPFS.

ION architecture diagram

Like HTTP, IPFS does not have user identity built into the protocol. However, IPFS provides resiliency, validation, and future-proofing features that HTTP cannot:

  • The content addressability of IPFS means that ION nodes who are pulling the CIDs off a blockchain or other underlying public network don’t need to care where the transaction data resides. This means they can switch servers or datacenters, or new storage nodes can come online without requiring any code or infrastructure changes and without the addresses needing to change.
  • ION nodes also don’t need to worry about the data being manipulated or tampered with, because hash-based addressing means the cryptographic verification of the data is built into the network request itself.

IPFS in ION

The ION implementation is in JavaScript (specifically, TypeScript) so it made sense for them to use js-ipfs as a Node.js service. ION rolls up batches of identity transactions, publishes it through their IPFS node, and then writes the address (CID) of that batch to the Bitcoin blockchain.

In order to meet Microsoft’s needs for using js-ipfs as a long-running process we added cancelable requests to all APIs, ensuring that as requests were being made and handled, the underlying objects, memory, file handles, and other resources created up and down the stack are cleaned up properly. Huge thanks goes to Alex Potsides (@achingbrain) for implementing this long-needed feature, which shipped in js-ipfs 0.44.0.

What this looks like for developers is the ability to set timeouts on requests:

const cid = new CID('QmWillNeverResolve')

try {
  await ipfs.get(cid, {
    timeout: 1000 // abort after 1000ms
  })
} catch (err) {
  console.err(err) // err is a TimeoutError
}

Try ION Now!

This is the public beta of ION, and it is now running live on the Bitcoin blockchain.

In the launch post, Microsoft’s ION project lead Daniel Buchner explains how to run a node and use decentralized identities in your apps and services today.

The project is open source, built on open standards, and you can run your own node—so try it out or contribute to the project today!

]]> IPFS Weekly 92 https://blog.ipfs.io/weekly-92/ Wed, 10 Jun 2020 00:00:00 +0000 https://blog.ipfs.io/weekly-92/ Welcome to IPFS Weekly #92 Here’s what’s happening in the InterPlanetary File System galaxy! 🚀 Join us next Wednesday for our June meetup Next week the IPFS community will be gathering for a virtual meetup on June 17th. Welcome to IPFS Weekly #92

Here’s what’s happening in the InterPlanetary File System galaxy! 🚀

Join us next Wednesday for our June meetup

Next week the IPFS community will be gathering for a virtual meetup on June 17th. Join us for a deeper dive into our latest releases, meet the IPFS core developers, and sign up to give a demo of your own!

Register to attend

js-ipfs 0.46.0 released

The JavaScript implementation of IPFS recently shared their latest progress on the IPFS blog. Goodies include Bitswap 1.2.0, much faster file transfers, and go-ipfs 0.5.x compatibility.

Read the release notes

Powering through the API gateway enabling IPFS and Filecoin data management

Earlier this month, Powergate, Textile’s API driven solution to deploying multi-tiered storage based on IPFS and Filecoin into systems and applications, shared some insight into the project’s development progress. Up next? The JavaScript Powergate Client.

Learn more about Powergate

Quote of the week

Brand new on IPFS

People build the coolest stuff with IPFS

  • This repository exists to accumulate and contextualize evidence of police brutality during the 2020 George Floyd protests, and has an open PR to create a public IPFS Follow Cluster.
  • IPDR is a Docker Registry tool that proxies Docker registry requests to IPFS for pushing and pulling images.
  • Git Large File Storage offers custom transfer and extension implementation that makes using IPFS for storage easy.
  • Check out IPIDE, an extensible IDE hosted on IPFS.

Job hunting? Work on IPFS!

Community Engineer, libp2p: Help introduce new open source projects to libp2p and clear hurdles for them to adopt the libp2p stack for the networking layer of their system. Protocol Labs, Remote

Engineering Manager, IPFS: Engineering managers have the challenging and exciting task of supporting and empowering engineering teams to deliver complex internet-scale systems in an environment defined by curiosity, passion, and a love for open source. Protocol Labs, Remote

IPFS Community Lead: Provide tactical support for the IPFS community, creating a global community strategy, and executing this strategy on the ground! Protocol Labs, Remote

Senior Software Engineer Rust/Go: You would be working with our distributed team of engineers on new products, core protocols or client projects. Most of the work we do is or will eventually be open source. Equilibrium, Remote

Software Engineer, the Underlay Project: Strong full-stack web developer with an interest in semantic web and decentralized web technologies. The Knowledge Futures Group, Remote

View our job postings

Take care of yourselves and each other. ❤️

Get involved with IPFS by checking us out on GitHub, joining discussions on our community forum, or hitting us up in chat. Have a suggestion? Email us.

Get the IPFS Weekly in your inbox, each Tuesday.

]]> libp2p in 2020 https://blog.ipfs.io/2020-06-09-libp2p-in-2020/ Tue, 09 Jun 2020 00:00:00 +0000 https://blog.ipfs.io/2020-06-09-libp2p-in-2020/ If you’re doing anything in the decentralized, peer-to-peer space, you’ve probably heard of libp2p - a modular networking stack for peer-to-peer applications. If you’re doing anything in the decentralized, peer-to-peer space, you’ve probably heard of libp2p - a modular networking stack for peer-to-peer applications. libp2p consists of many modular libraries from which p2p network developers can select and reuse just the protocols they need, while making it easy to upgrade and interoperate between applications. This helps web3 developers get up and running faster, build more resilient decentralized applications, and utilize advanced features like decentralized publish-subscribe and a distributed hash table.

What makes libp2p different from the networking stack of today is its focus on transport agnosticism, modularity, and portable encodings (like multiaddr). These properties make libp2p the networking layer of choice for most new dweb projects, blockchains, and peer-to-peer applications. Read more about why projects are choosing to build on libp2p, or watch the recent talk from Tech Lead Raul Kripalani at DevCon5.

2019 Growth

The past year has seen amazing improvements to libp2p with the protocol quickly becoming the de facto web3 networking layer for a huge swath of projects.

Probably the most significant news on that front is the official adoption of libp2p as the networking layer for the Ethereum 2.0 Network Specification. Ethereum 2.0 is the next generation of Ethereum, with a large and thriving network of clients, dapps, and developers. A big focus for libp2p in 2019 was leveling up stability, scalability, and ease of use to meet the requirements of such a large scale blockchain.

libp2p at DevConV

libp2p now has 7 language implementations:

The other major new blockchain building on libp2p is Filecoin which plans to launch their public storage and retrieval network this summer. With over 7.5 PB of storage already live across an active community of testnet miners, a big focus for libp2p in 2019 was landing the performance and hardening improvements to meet Filecoin’s needs.

In addition to Filecoin and Ethereum 2, there are now more than two dozen projects building directly on libp2p, including: Polkadot and Parity Substrate, Nervos, Status.im, Paratii, 0x, Coda, Enigma, Flow, Harmony, Golem, Keep Network and Validity Labs. Groups like Matrix.org are also in the early stages of adding libp2p support to their applications, helping their users benefit from the resilience and openness of p2p networking.

These newcomers to the libp2p ecosystem are a big deal - but they actually still make up less than half of all active libp2p nodes! The vast majority of libp2p end users come from dapps, tools, and projects using libp2p through IPFS, the InterPlanetary File System!

Millions of end users and over 100,000 libp2p nodes participate in the IPFS network every week. Organizations like Opera, Ethereum, Brave, Netflix, Microsoft, and Cloudflare have joined the IPFS ecosystem, and hundreds of nascent community-owned projects like Audius, Fleek, Textile, Qri, Pinata, OpenBazaar, Berty, Infura and 3Box are building their businesses with IPFS.

2019 and early 2020 features

The focus for the year was hardening and evolving the stack - and we blew that out of the water! Here are some of the features the core team and our community have contributed to libp2p:

  • In 2019, we added AutoRelay, a feature that ensures any user can launch a libp2p node easily without having to open ports on their router or do anything technically complicated. Relay nodes will handle the relaying of traffic. Coming soon is an automatic upgrade to direct connections from relayed connections.
  • We added better connection management, with decaying tags and connection gating.
  • We built and released Gossipsub, wrote a new explainer, and later enhanced with security extensions including peer scoring, opportunistic grafting, peer exchange, and other features.
  • Specs improvement in 2019 has made a night-and-day difference for cross language compatibility and the ease of implementing libp2p in new languages.
  • We simplified the go-libp2p codebase by centralizing abstractions and interfaces under the go-libp2p-core module, decommissioning 12+ repos along the way. Further codebase simplification is on the roadmap for 2020.
  • A huge focus for the year was the Async Await refactor in js-libp2p - enabling the use of streaming iterables instead of pullstreams, along with some interface refactors.
  • The libp2p core team has launched a massive documentation overhaul. Check out docs site and the new Getting Started guide.
  • We’ve completed a major DHT improvement project. Queries to the DHT are much faster, thus better supporting DHT-intensive applications like IPFS’s name resolver.
  • Support for Noise handshakes has been spec’ed, and is now available in all 7 implementations of libp2p.
  • We’ve continued to update our QUIC implementation to keep pace with new releases of the QUIC standard. We’ve also made progress on implementing QUIC in js.
  • We introduced certified peer records to enhance foundational protocol security. They’re used by gossipsub and the DHT, and are available to application protocols too.
  • We have prototyped an introspection protocol, with a reference implementation in go-libp2p. We’ll soon release a framework and UI shell that will interoperate with all libp2p implementations.

Releases

Community

We saw huge growth in the libp2p team and community this year, with presentations and workshops at a variety of conferences including EthCC 2, Web3 2019, EthBerlin Zwei, IPFS Camp, ReadyLayerOne, DevCon V (talks: state of libp2p, gossipsub), and more.

The libp2p project also launched a development grants program. The initial kickoff for the program at EthBerlinZwei led to several significant features being implemented by the community, including Noise handshakes in Go and Lua Wireshark dissectors. The bounties program is a great way to get involved in the libp2p project. Check it out on GitHub!

2020 H1 in review

Throughout the first half of the year, our top focus was supporting the needs of current and in-flight adopters, especially IPFS, Eth2, and Filecoin. So far, we have delivered on the following.

Content routing and connectivity

To support IPFS and all our other dapp users out there, we’ve improved libp2p’s content routing and connectivity to make finding and sharing content in distributed networks fast and reliable. This meant we needed to dive deep into major systems like the libp2p DHT, providing subsystem, NAT traversal, and more to optimize and refactor for performance. We’ve been collaborating closely with Protocol Labs’ IPFS and Testground teams to make this a reality. Read more about our progress in the IPFS 0.5 launch post, and our path to get there in the Road to the New DHT post!

Content Routing Speed

Launching Testground to the world

What is Testground you ask? Only the coolest new testing tool for large-scale p2p networks! When starting our network upgrade plans, it was clear that the scope and complexity of changes we’d need to make to core IPFS network functions would be impossible to optimize without powerful testing tools to simulate each change to network performance at scale. We couldn’t find a tool with the scale and configurability we needed, so we built the testing tool we needed, and made it modular and system agnostic just like libp2p so other p2p networks can take advantage of this new OSS infrastructure too!

Launching Testground

Gossipsub security enhancements

For our large blockchain users, the focus of Q1 was stability, scalability, and security hardening. We designed and implemented a new version for the Gossipsub protocol, v1.1. This new P2P PubSub router includes several security extensions that add a protection to Sybil, eclipse, and spam attacks.

This work is of extreme importance as our adopters (e.g., Filecoin and Ethereum 2.0) require a secure messaging layer to distribute their time sensitive and valuable data, without being vulnerable to malicious actors.

You can find the specification and implementation on GitHub. We will be releasing a report that fully describes how we identified the attack vectors, developed the motivations and went through implementing, testing, evaluating under multiple network conditions and got it audited. Stay tuned.

Gossipsub v1.1

Going forward

Both of these workstreams (content routing performance and gossipsub hardening) continue to be top priorities with IPFS having launched go-ipfs 0.5 (their largest network upgrade in years), and Ethereum and Filecoin gearing up for Q3 mainnet launches, as well as other adopters such as Polkadot (who went live recently!), Flow, Coda, 0x, Matrix, Keep Network, Enigma, Harmony, Golem, Validity Labs, etc., reaching new milestones.

Furthermore, in H2 we plan to work on protocol negotiation enhancements, a more efficient connection boostrapping flow, an implementation alignment framework (including a long-awaited conformance test kit), code refactors, and a lot more!

Growing the team

Protocol Labs’s small and scrappy libp2p core team transformed into a true force of nature with some great additions last year. Our doors will continue to be open for great people from both inside and outside of the ecosystem.

In particular, we’re looking for a new Project Lead to help grow and steward the libp2p project and ecosystem to new heights in 2020. Check out the role description for more specific details and information on where you can apply. In particular, we’re looking for candidates with proven leadership experience, a passion for p2p tech, and an interest in scaling the libp2p community as it becomes critical infrastructure for new, large users such as Ethereum (with launch of the Beacon chain) and Filecoin.

Excited about building the networking stack for a more resilient, upgradeable, open network for humanity’s knowledge? We’d love to get in touch! Apply, or just shoot mike@libp2p.io a quick email with your thoughts or questions!

Thank you

2019 was a fantastic, high growth year for the libp2p project and 2020 is already well on track to be even more exciting. Huge thank you to the thousands of community members who contributed to all the new language implementations, feature improvements, performance speedups, and amazing projects bringing libp2p directly to end users.

Thank you for your support! We’re excited to continue working with you to build libp2p.

Appendix

References:

]]> js-IPFS 0.46.0 released with new faster bitswap and go-IPFS 0.5 compat https://blog.ipfs.io/2020-06-08-js-ipfs-0-46/ Mon, 08 Jun 2020 00:00:00 +0000 https://blog.ipfs.io/2020-06-08-js-ipfs-0-46/ 🔦 Highlights Bitswap 1.2.0, transfer speed improvements & go-IPFS 0.5.x compatibility Blink and you’ll miss it, js-IPFS@0.45.0 js-IPFS@0. 🔦 Highlights

Bitswap 1.2.0, transfer speed improvements & go-IPFS 0.5.x compatibility

Blink and you’ll miss it, js-IPFS@0.45.0 js-IPFS@0.46.0 has been released; goodies include Bitswap 1.2.0, much faster file transfers, and go-IPFS 0.5.x compatibility!

↔️ Bitswap 1.2.0

Version 1.2.0 of the Bitswap protocol has landed in js-IPFS and brings a host of performance improvements around coordinating the supply of blocks to peers across the network. For example, peers can now respond to WANTs with messages saying they have the block but not sending it, in order to later send multiple blocks in one message instead of many small messages resulting in less chatty network operations and faster overall transfer times.

See ipfs/js-ipfs-bitswap#204 for more!

💨 Faster file transfers

The Bitswap module has had a bit of an overhaul and is now much faster at pulling blocks from other nodes across the network. Before, we waited a little while before sending our wantlist to newly connected peers—we’ve reduced that window and seen a massive speedup (smaller bars are better):

Graphs showing Bitswap speedups

As you can see, the new js-IPFS is about 3x faster at pulling large files from other nodes on the network than the previous release, and that increases to about 5x as the files get smaller.

We’re not stopping there though, there are plenty more performance improvements on the way!

See ipfs/js-ipfs-bitswap#224 for all the details.

🤝 go-IPFS v0.5.x compatibility

The new version of go-IPFS brought a whole slew of changes with it—check out our blog post for more details.

The good news is js-IPFS has taken on a whole heap of these features and the ipfs-http-client is now 100% compatible with go-ipfs@0.5.x. Your js-IPFS nodes will now happily chat away to go-IPFS nodes over the network via Bitswap and gossipsub too.

See ipfs/js-ipfs#3013 for the full rundown.

✨New features

  • Using an AbortSignal to cancel a request to get a block (via ipfs.dag.get or ipfs.cat for example) will now cause those blocks to be removed from your wantlist (ipfs/js-ipfs-bitswap#214)
  • ipfs.dht.get(key, [options]) accepts a string or a buffer as key (#3013) (0900bb9)

🦟 Bugs fixed

🏗 API Changes

Core API

  • ipfs.ls no longer supports a sort option (#3013) (0900bb9)
  • ipfs.key.gen defaults to 2048 bit RSA keys (#3013) (0900bb9)

CLI

  • ipfs files ls no longer supports the -U (unsorted) flag (#3013) (0900bb9)

HTTP API

  • POST /api/v0/pin/list no longer supports non-streaming responses (#3013) (0900bb9)

HTTP API Client

  • ipfs.dht.get resolves to a Buffer in line with the core API (#3013) (0900bb9)

🗺️ What’s next?

Check out the js-IPFS Project Roadmap which contains headline features organised in the order we hope them to land.

Only large features are called out in the roadmap, expect lots of small bugfix releases between the roadmapped items!

😍 Huge thank you to everyone that made this release possible

🙌🏽 Want to contribute?

Would you like to contribute to the IPFS project and don’t know how? Well, there are a few places you can get started:

⁉️ Do you have questions?

The best place to ask your questions about IPFS, how it works, and what you can do with it is at discuss.ipfs.io. We are also available at the #ipfs channel on Freenode.

]]> IPFS Weekly 91 https://blog.ipfs.io/weekly-91/ Wed, 27 May 2020 00:00:00 +0000 https://blog.ipfs.io/weekly-91/ Welcome to the IPFS Weekly Here’s what’s happening lately in the InterPlanetary File System galaxy and beyond! 🚀 The Road to the New DHT At the end of April, we released our largest update yet: go-ipfs 0. Welcome to the IPFS Weekly

Here’s what’s happening lately in the InterPlanetary File System galaxy and beyond! 🚀

The Road to the New DHT

At the end of April, we released our largest update yet: go-ipfs 0.5.0. This upgrade brought major performance and reliability improvements to IPFS — especially on the content discovery and routing front. We’d like to take you through our journey to re-write the DHT. Read the full story on the blog. 

js-ipfs 0.44.0 released 🎉

Last week, we shipped the latest release for js-ipfs, version 0.44.0! Meaning, you can now use AbortControllers to cancel requests in js-ipfs! No more waiting around to see if your request ever completes. Read the changelog post for full details on the release.

It’s true; Gossipsub v1.1 is here

The Gossipsub Task Force has been hard at work exploring and analyzing various attack vectors on public and permissionless messaging networks. With that knowledge, we’ve crafted and iterated on mitigating strategies to make our beloved libp2p PubSub Router work in adversarial environments. The result is Gossipsub v1.1.

Top highlights this week

Quote of the week

People are building the coolest stuff with IPFS

  • Zippie uses dynamic IPFS loading for a seamless mobile web experience.
  • Over 125,000 IPFS CIDs added to Cybers network as part of its incentivized tournament, Game of Links, in which users add IPFS CIDs to build a decentralized knowledge graph, which is used by the search engine
  • Fury is an experimental dependency manager and build tool for Scala
  • Textile updated their GitHub Actions, so the Textile Hub is back on the Github Marketplace!
  • An Introduction to Dwebsites built on IPFS
  • Sapien Wallet, built on Textile and IPFS
  • Fleek releases their new Storage SDK Cookbook

Join us at Distributed Camp 2020

This Friday through Sunday, members of the IPFS, Dat, I2P, WebTorrent communities and more, will gather to talk all things distributed for a special hands-on event: Distributed Camp 2020. Enjoy two days of immersive, hands-on workshops where you get to try out some of the latest web technologies for yourself and a Sunday Unconference where you get to learn of other projects in this space as well as show off what you made! Registration is free.

Are you using (or plan to use) js-ipfs in browser?

Then we want to hear from you! Your input would really help to shape the plan for changes that we hope will improve your and your users’ experience. Check out the discussion on reducing js-ipfs overhead by sharing node across browser tabs now!

Missed the 0.5 Meetup? Watch the videos now

Check out the full playlist of presentations from the event, from insights on content routing changes and improvements to the intricacies of Bitswap and subdomain gateways! And don’t forget to watch the lightning talks. Enjoy!

Take care of yourselves and each other. ❤️

Get involved with IPFS by checking us out on GitHub, joining discussions on our community forum, or hitting us up in chat. Have a suggestion? Email us.

Get the IPFS Weekly in your inbox, each Tuesday.

]]> js-ipfs 0.44.0 released https://blog.ipfs.io/2020-05-21-js-ipfs-0-44/ Thu, 21 May 2020 00:00:00 +0000 https://blog.ipfs.io/2020-05-21-js-ipfs-0-44/ 🔦 Highlights Cancellable requests and a new browser datastore js-IPFS@0.44.0 is out of the door with support for cancelling requests and a leaner, meaner datastore for the browser! 🔦 Highlights

Cancellable requests and a new browser datastore

js-IPFS@0.44.0 is out of the door with support for cancelling requests and a leaner, meaner datastore for the browser!

🤖 Cancellable requests

A user should be able to cancel long-lived asynchronous API operations. For example, if you are fetching the content for a CID from the network, and that CID is not resolveable, you should be able to set a timeout value for the request, after which the content would no longer be sought and control would be returned to your code with an error message describing what happened.

This is not as straightforward as it seems in JavaScript because a Promise is returned from the API call to resolve a piece of content and the Promises spec includes nothing about cancelling a Promise.

However, the browser fetch API has a notion of an AbortSignal which can be used to abort web requests. They are interacted with via an AbortController:

const controller = new AbortController()

setTimeout(() => {
  controller.abort()
}, 1000)

try {
  await result = fetch('http://example.com', {
    signal: controller.signal
  })
} catch (err) {
  console.err(err) // prints: user aborted the request
}

We’ve taken this approach and integrated it into every API call, so you can now use AbortControllers to cancel requests! We’ve also used this mechanism to add a timeout option to every API call which internally uses an AbortSignal to stop the request and throw a TimeoutError after the specified amount of time:

const cid = new CID('QmWillNeverResolve')

try {
  await ipfs.get(cid, {
    timeout: 1000 // abort after 1000ms
  })
} catch (err) {
  console.err(err) // err is a TimeoutError
}

You could also roll this yourself:

const cid = new CID('QmWillNeverResolve')
const controller = new AbortController()

setTimeout(() => {
  controller.abort()
}, 1000)

try {
  await ipfs.get(cid, {
    signal: controller.signal
  })
} catch (err) {
  console.err(err)
}

The AbortSignal is passed all the way down the call stack to the libp2p and ipld components that underpin IPFS. In this initial release, timeouts are supported by the IPFS layer and in future releases libp2p and ipld will use the AbortSignal to also free up resources and perform any other necessary cleanup that would have been done if the request had completed successfully.

New browser datastore

js-IPFS@0.44.0 brings a new datastore to the browser. In the browser, all blocks and other repo data is currently stored in IndexedDB, given that it is the only way available to persist and query large amounts of user data at the time of writing.

In node and go-ipfs, LevelDB has been the datastore of choice for application data (though blocks have been stored on the filesystem) and go-IPFS is moving to Badger).

Out of this grew interface-datastore — a spec for storing key/value pairs implemented by the datastores used by IPFS. In the browser this was datastore-level which was backed by level-js, which was backed by IndexedDB.

Phew.

To remove a few layers here, we’ve created the datastore-idb which implements the interface-datastore spec, backed by IndexedDB without going via level.

What does this mean?

The new datastore is smaller and faster and the upgrade is seamless since it’s still IndexedDB under the hood:

datastore-idb up to 44% faster and 69% smaller

batch idb x 5.26 ops/sec ±4.57% (29 runs sampled)
batch level x 2.92 ops/sec ±3.76% (19 runs sampled)
Fastest is batch idb

size 37.32KB to 11.41KB

✨New features

  • Cancellable API calls (#2993) (2b24f59), closes #3015
  • ipfs.block.put now accepts the pin argument in line with go-ipfs #3015

🦟 Bugs fixed

🗺️ What’s next?

Check out the js-ipfs Project Roadmap which contains headline features organised in the order we hope them to land.

Only large features are called out in the roadmap, expect lots of small bugfix releases between the roadmapped items!

😍 Huge thank you to everyone that made this release possible

🙌🏽 Want to contribute?

Would you like to contribute to the IPFS project and don’t know how? Well, there are a few places you can get started:

⁉️ Do you have questions?

The best place to ask your questions about IPFS, how it works, and what you can do with it is at discuss.ipfs.io. We are also available at the #ipfs channel on Freenode.

]]> Gossipsub v1.1 brings hardening extensions to PubSub https://blog.ipfs.io/2020-05-20-gossipsub-v1.1/ Wed, 20 May 2020 00:00:00 +0000 https://blog.ipfs.io/2020-05-20-gossipsub-v1.1/ The gossip you’ve heard on the streets is correct, Gossipsub v1.1 is here and it packs several security hardening extensions along with new testing, documentation, and an updated specification.

The gossip you’ve heard on the streets is correct, Gossipsub v1.1 is here and it packs several security hardening extensions along with new testing, documentation, and an updated specification.

The Gossipsub Task Force has been hard at work exploring and analyzing various attack vectors on public and permissionless messaging networks. With that knowledge, we’ve crafted and iterated on mitigating strategies to make our beloved libp2p PubSub Router work in adversarial environments. The result is Gossipsub v1.1 🚀

If you are new to libp2p PubSub and especially Gossipsub, we recommend checking out the Gossipsub v1.1 talk at the Matrix Virtual Meetup. It will give you the complete motivation and background of libp2p PubSub and introduce you to Gossipsub v1.1.

🔍 What is Gossipsub v1.1

The main focus for Gossipsub v1.1 is security. In addition, to the speedy message propagation guarantees from v1.0, this new version features several hardening extensions that make Gossipsub more resilient to a wide range of attacks. Some highlights include:

1) Peer scoring: Peers now monitor their directly-connected neighbours using a score function that reflects the peer’s usefulness. Well-behaving peers are kept in the mesh, while badly-behaving ones are dropped in order to protect the network from malicious actors.

2) Adaptive gossip dissemination: In this new version, you can adjust the gossip factor to control the number of peers your node gossips with. This enables you to increase/decrease the amount of gossip, while keeping a baseline.

3) Opportunistic Grafting: A new mechanism used to cherry pick well-behaving peers that are currently not part of a node’s mesh. Those well-behaving peers are then opportunistically inserted into the mesh to improve the median score of participating peers.

4) Prune Peer eXchange: Prune Peer exchange gives a node a set of recommended peers to connect when it gets dropped from another peer’s mesh, helping reduce the dependency on ambient peer discovery mechanisms.

5) Extended Message Validators: You can now add your custom message validator so that you can instrument Gossipsub to recognize (accept/reject) which messages are valid/invalid within the context of your application.

6) Outbound Mesh Quotas: In v1.0 mesh peers are randomly selected, without any weight given to the direction of the connection. In contrast, v1.1 implements outbound connection quotas, so that a peer tries to always maintain a number of outbound connections in the mesh.

The blend of these security measures, together with the properties of the “gossiping mesh” check the two most important boxes for message propagation protocols in permissionless networks, namely: i) fast message propagation, ii) security against attacks that attempt to degrade the quality of the service provided by the network.

All of these upgrades can be consulted in the Gossipsub v1.1 specification, which if you ask us, it is a joy to read!

Additionally, with the new release, the reference implementation of Gossipsub got additional documentation and test coverage.

🏎 Give Gossipsub v1.1 a test ride

Chat application example

We’ve prepared a brand new example for you to give the latest Gossipsub a try. In this example, you get to experiment with a chat application powered by libp2p. It will enable you to join different channels by subscribing to PubSub topics and see other peers joining the room as they subscribe to the topic.

You can find the code for this example at https://github.com/libp2p/go-libp2p-examples/tree/master/pubsub/chat

Test Gossipsub’s performance with Testground

We are also releasing a brand new libp2p Test Plan that uses Testground to benchmark Gossipsub v1.1.

With this Test Plan, you can run an emulation (yes, it will run real Gossipsub nodes!) of a Gossipsub Network in which you can adjust:

  • The number of nodes
  • The latency of each link and the jitter %
  • The length of the run
  • The bandwidth available to each link

To play with this Test Plan, check the instructions at https://github.com/libp2p/test-plans/tree/master/pubsub.

🎁 We are preparing something more for you

We’ve worked with networking, P2P, and security researchers to create this iteration of Gossipsub v1.1 and we’ve been cooking a comprehensive analysis report to share with you on our approach and how we evaluated our mitigation strategies. We are shooting to share this report in June, stay tuned!

That’s it for now. Hope you enjoy the release and happy gossiping!

The Gossipsub Task Force - David Dias, Dmitris Vyzovitis, Yiannis Psaras, Yusef Napora, Dirk McCormick

]]> IPFS Weekly 90 https://blog.ipfs.io/weekly-90/ Wed, 20 May 2020 00:00:00 +0000 https://blog.ipfs.io/weekly-90/ Welcome to the IPFS Weekly Here’s what’s happening lately in the InterPlanetary File System galaxy and beyond! 🚀 The path to decentralization and go-ipfs 0. Welcome to the IPFS Weekly

Here’s what’s happening lately in the InterPlanetary File System galaxy and beyond! 🚀

The path to decentralization and go-ipfs 0.5 on the Zero Knowledge Podcast

IPFS Project Lead Molly Mackinlay was recently a guest on the Zero Knowledge Podcast talk about her experience guiding the IPFS project, as well as libp2p, Bitswap, Testground, the use cases and projects working with these tools, and the specific upgrades made with the recent 0.5 IPFS release. Enjoy!

Experience the IPFS Pinning Summit, again!

True, the IPFS Pinning Summit has come and gone, but if you missed it, we have you covered. Watch all of the videos from the event and be sure to read our recap from the two-day summit.

Are you using (or plan to use) js-ipfs in browser?

Then we want to hear from you! Your input would really help to shape the plan for changes that we hope will improve your and your users’ experience. Check out the discussion on reducing js-ipfs overhead by sharing node across browser tabs now!

Top highlights this week

Quote of the week

People are building the coolest stuff with IPFS

Have you shared your thoughts on IPFS GUI Tools?

In order to help prioritize future improvements to IPFS tooling and interfaces, we’d appreciate learning more about how the IPFS Desktop app and the in-browser IPFS GUI (also known as Web UI) fit into your overall workflow. This survey should take no more than 5 minutes, and will make a big difference—so thank you in advance for your time!

Job hunting? Work on IPFS!

  • Community Engineer, libp2p: Help introduce new open source projects to libp2p and clear hurdles for them to adopt the libp2p stack for the networking layer of their system. Protocol Labs, Remote
  • Engineering Manager, IPFS: Engineering managers have the challenging and exciting task of supporting and empowering engineering teams to deliver complex internet-scale systems in an environment defined by curiosity, passion, and a love for open source. Protocol Labs, Remote
  • IPFS Community Lead: Provide tactical support for the IPFS community, creating a global community strategy, and executing this strategy on the ground! Protocol Labs, Remote
  • Senior Software Engineer Rust/Go: You would be working with our distributed team of engineers on new products, core protocols or client projects. Most of the work we do is or will eventually be open source. Equilibrium, Remote.
  • Software Engineer, the Underlay Project: Strong full-stack web developer with an interest in semantic web and decentralized web technologies. The Knowledge Futures Group, Remote

Take care of yourselves and each other. ❤️

Get involved with IPFS by checking us out on GitHub, joining discussions on our community forum, or hitting us up in chat. Have a suggestion? Email us.

Get the IPFS Weekly in your inbox, each Tuesday.

]]> The Road to the New DHT https://blog.ipfs.io/2020-05-19-road-to-dht/ Tue, 19 May 2020 00:00:00 +0000 https://blog.ipfs.io/2020-05-19-road-to-dht/ At the end of April, we released our largest update to go-ipfs to date: IPFS 0.5.0. This upgrade brings major performance and reliability improvements to IPFS — especially on the content discovery and routing front.

At the end of April, we released our largest update to go-ipfs to date: IPFS 0.5.0. This upgrade brings major performance and reliability improvements to IPFS — especially on the content discovery and routing front. These gains are brought to you largely by our rework of the Distributed Hash Table (DHT) which powers most peer and content discovery in the IPFS Public Network.

We’d like to take you through our journey to re-write the DHT - from identifying the underlying issues, to how we approached designing a solution, to ensuring it would work in practice. Our work led to major performance gains, as well as a new development process and Testground, a whole new tool for large-scale network testing (read more).

If you want to make use of these new improvements, please upgrade IPFS now!

First a refresher: What’s the DHT?

The DHT, or distributed hash table, acts like both a catalog and navigation system for IPFS, helping the network keep track of and find data. A regular hash table is a key-value store where the keys are hashed and stored in one place. In the case of IPFS, the keys are the Content Identifiers (CIDs) of a block of data, and the values are the set of peers who have each block.

The DHT distributes and stores these pairs of keys and values in small tables across many nodes or peers throughout the network. Peers, in turn, store information about where to physically locate specific content. Kademlia, our DHT implementation, helps keep track of which nodes are providing what data to others in the network.

In theory, the DHT should make finding, providing, and fetching data among nodes a well-orchestrated and very efficient affair. However, throw in real world conditions and a rapidly scaling network, and things can go awry.

Let’s rewind

The IPFS Public Network experienced substantial growth last year, quickly scaling 30x, with hundreds of thousands of nodes now participating in the network! Major software deployments and new decentralized apps came on board with expectations for a production-ready network with the reliability and performance to match. Many were using IPFS as a decentralized CDN (Content Delivery Network) to share and find content across the network. Unfortunately, due to a proliferation of new nodes unable to route peers to the desired content, the network was letting some of these users down.


In early 2019, we started to hear rumblings around content routing issues, which were exacerbated as more and more nodes joined the network over Q1 and Q2 - many of them behind home firewalls or NATs. We were constantly bootstrapping and patching the network to remedy performance and reliability issues. But, we couldn’t really test the efficacy (or truly diagnose any regressions) of these patches until they were in production, making it hard to iterate and try out larger fixes to network configuration.

We clearly had a problem, and we were hearing about it from the community. Our first step was to investigate thoroughly to determine what fixes we should prioritize first, and identify the tooling we needed to land our changes successfully.

Narrowing in

After some investigation, it was clear that our DHT implementation, combined with the recent growth of undialable nodes, was the culprit behind the slow, unreliable content routing. We had a chance to investigate this more deeply while we were together at IPFS Camp, and discovered a few problem areas:

  • Peer Availability: As new peers joined the network we treated them equally, but most couldn’t actually be reached because they were behind firewalls or NATs. DHT query time was wasted trying to dial these peers that never got requests.
  • Query Termination: Even after a DHT query reached its target, it didn’t stop - it continued querying additional peers to make sure it had reached its target. While robust to the many new undialable nodes in the network, this process was very time consuming.
  • Routing Table Maintenance: Routing tables were clogged with many undialable peers, resulting in searches that were linear in the number of peers they queried instead of the expected log-scale efficiency.

Focus and go

With this analysis and the feedback synthesized from our core users and community, it was clear we needed to hone in on content routing performance as a core focus for the IPFS Project.

This meant some hard calls on prioritization: focusing our working groups on a targeted set of improvements to the DHT and routing logic to ensure we resolved these underlying performance and reliability concerns. In turn, we adjusted our product roadmap and stood up new core working groups, including one focused strictly on content routing and another on a new test infrastructure: Testground.

Test, test, test

We’d learned from our early patches that a network of our size needs to be able to test changes in a close to real-world setting to be able to make improvements quickly and reliably. We had the means to do unit tests and small simulations that could prove out basic tweaks, but no way to replicate thousands of nodes or typical network configurations in order to benchmark major changes. This left us without real data on the true impact of changes until they had gone live.

We had to change this quickly to be successful in righting the DHT. We formed a working group to remedy this gap; Testground was born. Testground is a platform for testing, benchmarking, and simulating distributed and peer-to-peer systems at scale, allowing us to validate the needed fixes and pressure test the broad DHT changes before IPFS 0.5.0 went live.

Testground gave us the tools to do data-driven development in our DHT rewrite. It also helped us in other parts of the IPFS 0.5.0 release: like prototyping new potential solutions, comparing potential fixes against each other, and stress-testing the edge cases and upgrade process. The repeatability and introspection Testground gave us was critical to accelerating our development velocity to reliably improve IPFS.

With the launch of Testground v0.5, we’re excited to share this tool with others as well to benchmark and evaluate their p2p networks. Start using Testground today!

Now for the “fix”

With Testground, we were able to benchmark and quantify our DHT issues around peer availability, query termination, and routing table maintenance. We started with simulations of the existing network - showing how our previous query logic interacted with evolving network parameters, like many nodes being undialable.


Testground allowed us to tweak these parameters to measure which changes would have the most impact on network performance. As we gathered for a Research Summit in January to compare potential mitigations to the ongoing network challenges, we were now empowered with a quantitative understanding of the network structure and the tools to immediately prototype and test out the resulting hypotheses from our discussions.

What followed was a whirlwind of test-plan writing to measure each change to the DHT logic in a real-world network setting. We tested our new systems for diagnosing undialable peers and removing them from our routing tables, measured success rates for ending our queries earlier, and even re-benchmarked our improvements to Bitswap to find additional improvements. Throughout these refactors, we were able to continually validate the performance of these changes with our Testground simulations, as well as a few well-placed canary nodes to run benchmarks against the full public network.

With these tools in hand, we sprinted 🏃‍♂️💨 from the summit to our release to make it all happen. From identifying the problem with the community, to scoping potential solutions, building testing tools, learning from past research, and executing on the needed technical fixes - this marathon of a release certainly put us through our paces - resulting in an epic list of new and improved features.

Up next, in Part 2 of this DHT Deep Dive, we’ll walk you through exactly what’s new with the DHT and go even deeper on the improvements we saw. In the meantime, if you haven’t upgraded yet, please do. We’ll all reap the benefits of a more performant DHT as more and more nodes update!

Learn more

]]> IPFS Pinning Summit Recap https://blog.ipfs.io/2020-05-13-ipfs-pinning-summit-recap/ Wed, 13 May 2020 00:00:00 +0000 https://blog.ipfs.io/2020-05-13-ipfs-pinning-summit-recap/ The first-ever IPFS Pinning Summit brought together IPFS pinning services, infra providers, and users for 2 days of presentations & meetings to discuss the current state of IPFS infrastructure, services, and tools - and the future opportunities in this space.

The first-ever IPFS Pinning Summit brought together IPFS pinning services, infra providers, and users for 2 days of presentations & meetings to discuss the current state of IPFS infrastructure, services, and tools - and the future opportunities in this space.

Day 1 focused on the accelerating growth of the IPFS ecosystem - especially the adoption and improvement of IPFS tooling and support for verticals like Web3 applications, DevOps pipelines, and Large Data on IPFS. The day also included presentations from major IPFS pinning services like Infura and Pinata, tools smoothing the IPFS upgrade UX like Fleek and UnstoppableDomains, and applications building on top of IPFS with strong use cases for IPFS infrastructure and tooling like Ceramic, Ucam, and Filecoin.


IPFS Project Lead Molly Mackinlay called the next 18 months a “critical period” for the IPFS community. “There are lots of key applications and projects being built, improving the ecosystem, and all kinds of developer tools are arriving that are transforming how applications and users of IPFS end up using the system,” Mackinlay said. “All of these developer tools and systems are changing the nature of how people store and move data.”

The Summit also welcomed a host of IPFS collaborators to the stage, who demoed their work and shared tips on running dweb infrastructure and techniques for improving these services. Here are a few ecosystem highlights:

  • Textile’s Andrew Hill and Aaron Sutula debuted Powergate, an API-driven solution for deploying multitiered storage across Filecoin and IPFS. Powergate gives clients easier access to control who and where their data is stored. Hill also detailed Textile’s ThreadsDB and Buckets, technologies that get us closer to a more intuitive object storage experience. Combining these tools with IPFS enables new pinning capabilities that support dynamic datasets and IPNS addresses. Watch both videos: on Powergate and on ThreadsDB & Buckets.
  • Infura’s Mike Godsey walked through the history and role of IPFS in Infura’s Web3 dev suite. Infura is leading the charge on permissioned gateways. Godsey also hinted at what’s next for Infura when it comes to building on the Filecoin network. “We are actively in discussions with the PL team about all of the interesting ways we can utilize Filecoin inside our infrastructure. We’re exploring specific Filecoin functionality that would be separate from our IPFS functionality. This is still in progress, but it’s super important to us.” Watch the video.
  • Brett Shear, co-founder and COO of Fleek, showcased how to use Fleek’s new IPFS/S3 product to upload, pin, and fetch files to/from IPFS at lightning speed. Watch Shear’s presentation.
  • IoTeX Co-founder Raullen Chai talked about how IoTeX, a blockchain solution for powering the Internet of Things, is teaming up with IPFS to create decentralized privacy and user data ownership with home security camera Ucam. Watch the video.
  • Pinata CEO Kyle Tut dove deep into data permanence and discussed how to keep the data behind a CID alive. Likening data preservation to the responsibility artists have to maintain their own work, Tut asked “Is Da Vinci responsible for maintaining the Mona Lisa? And the answer to that is, no! Da Vinci is no longer with us, the painting is something like over 500 years old, and the person or the body that takes care of that painting today is the Louvre Museum.” Similarly, ownership of digital data, Tut argued, can transfer as a token, from one wallet to another. It’s the file owner’s responsibility to pin that data on a network like IPFS to make sure it’s secure. Watch the video.



Day 2 focused on the future - with Juan Benet, creator of IPFS & Filecoin, highlighting new opportunities and improvements coming to the space, including a unified IPFS Pinning API, deeper integration of network services into IPFS tools like IPFS-Desktop and WebUI, and new capabilities like Powergate to bring Filecoin persistence to IPFS pinning services.

The IPFS Pinning Summit brought together 100+ participants for product demos, deep-dives into ecosystem progress, and lightning talks on new tools. Talks dived into how infra providers could tune their services to enhance growing verticals like video, gaming, or browsers - or how they might offer new, specialized products for private networks, access control layers, or network introspection.

Presentations also explored new opportunities related to the upcoming Filecoin Network launch - which adds an incentive layer to distributed data storage in IPFS. Filecoin Leads Pooja Shah and Juan Benet shared a number of new ways to integrate Filecoin to augment and enhance existing web3 infrastructure - like using Textile’s Powergate to create Filecoin-backed IPFS Pinning Services.


Watch all of the sessions here, including lightning talks on Filecoin’s GUI, Git on Threads, the Filecoin Discover Project, and many more.

Thanks for an amazing summit, and looking forward to the next wave of adoption and growth we create together! 🚀

]]>