RWKV-6 model: Finch 7B World 3

Now trained with an expanded and improved multilingual dataset, the latest Finch World 3 is the most capable 7B parameter class RWKV model yet! And you can use it today from either HuggingFace or the ChatRWKV inference runtime

Our goal is always to provide high-quality open-source AI models for everyone worldwide, regardless of nationality, language, or economic status. The RWKV architecture is designed to help reduce our impact on the environment, using a fixed amount of power per token regardless of context length. We invite interested developers to help us shape its future on the RWKV Discord server

Eval and benchmark

We tested Finch 7B World 3 using the EleutherAI lm-evaluation-harness across various typical industry benchmarks. Downstream performance improved significantly, now strongly beating Llama2 7B (trained on 2 trillion tokens) and closing in on Mistral 7B v0.1 and even Llama3 8B. We had theorized that the total tokens trained was the major difference between RWKV-6 models and modern Transformers, and seeing the continued performance improvement from further training reinforces that view. Llama3 8B is a larger model trained on 15 trillion tokens - nearly five times as many as Finch World 3 - and yet, the scores are close!

We’re looking forward to sharing the upcoming results from our new RWKV-7 architecture “Goose”, which may finally match or eclipse the modern transformer on a tokens-trained basis.

You can find the Finch architecture details in the Eagle and Finch research paper, recently presented at the Conference on Language Modelling.

Finch 7B World 3 has now been trained on a total of 3.1 trillion multilingual tokens. The training was accomplished in two steps: First, the original 1.1 trillion token Eagle (RWKV-5) checkpoint was upgraded to Finch (RWKV-6) and trained up to 1.4 trillion tokens with an expanded World v2.1 dataset. Then, the dataset was expanded again and training was continued for up to a total of 3.1 trillion tokens.

We added the following dataset (in addition to the original World 2 dataset details listed in the Eagle and Finch research paper) for the World V3 dataset.

Added in World v2.1

• cosmopedia
• adjustments to slimpajama inclusions
• dolma v1.6 reddit 
• Magpie-Align
• glaiveai_glaive-code-assistant-v3 
• cognitivecomputations_SystemChat-2.0_SystemChat 
• migtissera_Tess_tess-v1.5 
• openbmb_UltraInteract_sft 
• m-a-p~Code-Feedback~Code-Feedback

Added in World v3

• fineweb-edu 
• DCLM
• cosmopedia-v2 
• Buzz-V12 
• WebInstructSub 
• SKGInstruct 
• math-ai 
• TemplateGSM 
• all of starcoder
• python-edu (in HuggingFaceTB/smollm-corpus)

For the upcoming RWKV-7 “Goose” training runs, we will be improving and expanding the tokenizer to efficiently handle more world languages, and adding even more new dataset components.

Try out Finch World 3 today!

Acknowledgments

A big thank you to the following groups, who were instrumental in the continued development of the RWKV architecture and models:

• Recursal AI for its commitment to providing resources and development for the RWKV ecosystem - you can use their featherless.ai platform to easily run RWKV and compare to it, other language models

• EleutherAI for support and guidance, especially on benchmarks and publishing research papers about the RWKV architecture

• Linux Foundation AI & Data group for supporting and hosting the RWKV project

And of course, a huge thank you to the many developers around the world working hard to improve the RWKV ecosystem and provide environmentally friendly open-source AI for all.

Or more specifically, “windows 11: version 23H2” and “windows 10: version 22H2” …

C:\Program Files\Microsoft Office\root\vfs\ProgramFilesCommonX64\Microsoft Shared\OFFICE16

Today, you can literally walk into your local IT store, find a laptop with Windows 11 copilot, and search rwkv (enable system files), and find the files there.

With an estimated, half a billion, windows 11, and 1 billion windows 10 installations. This marks the largest rollout for RWKV in terms of installation 🤯

Is it real?

To validate the binaries, we have since decompiled them, to verify that they are based on the RWKV.cpp project, supporting up to version 5 of our models (we are currently on version 6).

So yes, it is real.

Our project is Apache 2 licensed, Microsoft is allowed to do this. (assuming proper Apache 2 license attribution)

What is Microsoft using it for?

While it’s unclear what Microsoft is specifically using our models for, it is believed, this is in preparation for local Co-pilot running with on-device models

RWKV's biggest advantage is its ability to process information like a transformer model, at a fraction of the GPU time, and energy cost. Making it one of the world’s greenest model

RWKV is probably used in combination with the Microsoft phi line of models (which handles image processing), to provide

• best-in-class multi-lingual support

• low computation, batch processing in the background (MS recall)

• general-purpose chat (though this is probably the phi model)

Its main advantages are its low energy cost and language support.

Fingers crossed on the rollout

For now, until the roll-out of offline co-pilot into the Microsoft operating system and/or Office 365. We will be keeping tabs, to see how our models are deployed into Windows.

We are excited to see what is next, as we scale out the deployment for the RWKV open source foundation model.

Change note: the article was originally citing 0.5 billion which is the estiamted size of windows 11 deployment.

It has been updated to include windows 10, as we have gotten confirmation for it as well.

Announcing the latest RWKV model: Finch 14B!

Finch is the 6th and latest version of the RWKV architecture, succeeding the Eagle / v5 lines of models. Finch improves upon Eagle by introducing data-dependence into the token shift and time-mixing, making Finch more efficient in managing its “long-term memory” as it processes a prompt, thereby giving it better range.

The Finch architecture is covered in detail alongside Eagle in https://arxiv.org/pdf/2404.05892 and Finches smaller than 14B have been appearing throughout the 2024 calendar year, with 14B representing the largest Finch trained to date (also the largest RWKV model - 7B was the maximum size trained of Eagle).

Training details and Evals

Both Finch 7B and Finch 14B are derived from continuing training of the Eagle 7B weights on the same dataset (known as World v2.1, the constituents of which are described here). The 14B model is derived from stacking two copies of the 7B model. Stacking effectively increases the short-term memory of the model (i.e. how much of the exact prompt feeds into the NN layers at each level) which has a different effect than widening the model.

We evaluated the Finch models using https://github.com/RWKV/lm-evaluation-harness, a fork of the standard LLM evaluation framework which also powers HuggingFace’s Open LLM Leaderboard (fork only to make the harness work via automation).

We ran a wide variety of benchmarks (235 in total), attempting to maximize breadth, while managing computation time (each of the models took 2 days to eval in our setup (!)).

Finch 7B improved +5.38% across all benchmarks while Finch 14B improved an additional +7.14% across all benchmarks (both figures relative to Eagle 7B). Given that Eagle 7B was the starting point for training for both models, the fact that there was increase is a given; the amount of increase is evidence of the value of Finch’s architectural changes, as well as that the depth of the model is not saturated by our data-set run (of 1.42T tokens).

If we focus specifically on the Open LLM Leaderboard v1 benchmarks, we see

Contributing GPU cluster time to RWKV!

RWKV is an open source project recognized by the Linux Foundation. There are various bottlenecks to the project, but GPU time is one of them, and we gratefully accept donations. If your organization has idle time please reach out at eugene@rwkv.com or nathan@rwkv.com to explore a donation and learn about what kinds of training runs those spare cycles could power.

References

• Model weights:

  • 14B

  • 7B

  • 3B

  • 1.6B

• Hosted inference: https://featherless.ai/models/RWKV/Finch-14B

• Training code: https://github.com/BlinkDL/RWKV-LM

EagleX v2 - in short

We extended the training of the previous Eagle 7B from 1.1 Trillion tokens to 2.25 Trillion tokens.

• A continuation based on the original Eagle 7B model,
  and the EagleX 1.7T model

• Ranks as the world’s greenest 7B model (per token)

• Trained on 2.25 Trillion tokens across 100+ languages

• Outperforms all 7B class models in multi-lingual benchmarks

• Passes LLaMA2 (2T) in multiple English evals, approaches Mistral (>2T?)

• All while being an “Attention-Free Transformer”

We are releasing RWKV-v5 Eagle v2, licensed under Apache 2.0, which can be used personally or commercially without restrictions.

• Download from HuggingFace

• Try it online today on: recursal.ai cloud platform

• Try on: our HF gradio demo

• Use our reference pip inference package, or any other community inference options (Desktop App, RWKV.cpp, etc), and use it anywhere (even locally)

• Fine-tune using our Infctx trainer

• [Pending PR] Get support merged into Huggingface transformers!

• All eval data can be found in the google sheet here

Building on bold claims

The original EagleX 7B 1.7T, trained by Recursal AI, made history as the first sub-quadratic model, to pass llama2 7B 2T on average in English eval.

Today we are releasing the 2.25T trained variant, which furthers the gap with the llama2 model.

The following report follows the same general format of the 1.7T model release, in eval details - to make direct comparision easier.

Winning English Perplexity

We start with the basics: Perplexity. Which is the loss value against the test dataset (lower score = better), i.e. how good the model is with the next token prediction.

In a major first, the EagleX model - now passes mistral in perplexity. And takes the lead in the 7B model weight class.

Leading Multi-lang Perplexity & evals

EagleX maintains the lead for best-in-class multi-lingual performance, with the incremental improvements we’re making to the Eagle line of models.

Most of the tasks here are common sense reasoning tests of a wide variety of formats, across languages including 23 of the world’s most widely used languages.

For the remaining languages, we urge the community to test and judge them themselves, over 100+ languages were trained. Over time, we would want more languages to be added to evals.

Going big on eval data

As per the previous 1.7T model, we ran ALL the benchmarks in EleutherAI `lm-eval-harness`, at commit `f78e2da`, with the following limitations:

• It has to be completed in under 30 minutes on 8x4090 (we were running lots of evals)

  • This rules out some of the rather more expensive long chain of thought evals

• We excluded all the personality/alignment evals

• Eval has to be executable across a wide variety of models, via lm-eval-harness

• All evals are 0 shot (no 5 shot-ing an MCQ question)

• We limited comparison to other models within the 7B weight class

These resulted in running 60+ major eval groups, which generated over 1,000+ data points per model. A data point count so high, that we had to drop standard error deviations, just to ensure the raw CSV file can be loaded in MacOS numbers.

21 English Evals

However, because 180+ evals are overwhelming, let’s first reduce down to 21 of the arguably most popular English evals, such as Lambada, Glue, Swag, Winogrande, TruthfulQA, MMLU:

Narrowing it down to the models that most of us actually care about - LLaMA, Mistral, EagleX, and Eagle-7b - the new EagleX v2 model outperforms LLaMA-2-7b on average across the 21 evals, and lags not far behind Mistral.

The Good

Now, let’s look at where our model is blowing the rest of the models out of the water.

First, the big stand out is the following 5 evals, in which both our 1.7T and 2.25T models beat even mistral 2T++ trained model (glue, anli, mmnli, swag), across multiple tasks focused around either contextual-based simple Q&A with common sense reasoning, or deductive logic.

PS: The jump for glue/mnli was high enough, that we needed to check the dataset specifically for contamination. Which we were not be able to find any. This jump is currently being attributed to multiple training datasets, along with data augmented / machine rewritten instruct dataset following a similar structure.

EagleX 2.25T, also performs better than LLaMA-2-7b in, lambada next token prediction and more importantly …

winograde, wnli, truthfulqa evals, which imply that the EagleX model would be applicable in RAG use cases, which are mainly contextual Q&A, with the right prompt engineering.

The Mixed

Next: the eval sets with mixed results.

For logiqa, we have very similar evals with 2 major variants. The results between EagleX and LLaMA are close enough, that it’s hard to say which model is clearly better between the two for these evals.

Similarly, sciq got slightly worse between 1.7T to 2.25T, but in general, all models are within trading blows of each other at 90%+ scoring.

The “Not too bad“ and the “Really Bad”

These are the evals the EagleX model performs worse when compared to both Mistral and LLaMA. However, for the evals that we’ve lost to LLaMA, it’s typically by a narrow margin.

One major error that occurred in the 1.7T model, was the accidental exclusion of the math dataset, which caused a degradation of math performance.

Since then, we have added back math text and math materials. Which boosted the arithmetic score. However, given the number of tokens between 1.7T and 2.25T, and the learning rate, the increase in math score was limited.

Our recommendation still stands that realistically IMO - no one should be depending on a 7B model for math (just saying)

180 English Evals

As per the previous 1.7T release. let’s zoom out, and look at it holistically across 180 English evals.

You can view the full results here

Although using the overall averages across all the evals does have a bias on the results towards larger eval sets (due to double counting, e.g. mmlu overall and many individual mmlu test), it does not change the ranking among the EagleX, Mistral, LLaMA and the original Eagle models.

However, these results are useful for smaller insights (as per the previous model as well). Such as measuring specifically gaps in knowledge by “subject domain” within the models.

Perhaps a good dataset + Scalable architecture:
is all you need?

The RWKV Open Source Foundation's goal is to ensure AI access is made accessible to everyone in the world, regardless of language or economic status.

In line with our goal, it does repeat the question. If the exact architecture, matter less than the data for the model performance?

If true, perhaps we should seek more efficient and scalable architecture, to increase accessibility for everyone regardless of language or economic status.

All while going beyond the English language, which represents only 17% of the global population.

And reducing the impact on our environment.

What’s next for the RWKV group?

This release marks the final release of the Eagle line of RWKV models. With the finalization of the v6 architecture as outlined in the paper here - https://arxiv.org/abs/2404.05892

Our next step is to move onto the v6 Finch line of architecture, which we expect to bring an incremental improvement on the v5 Eagle architecture.

This is made in consideration, that upcycling from Eagle to Finch line of models works from our existing experiments.

Roadmap

• v6 Finch: 0.1B, 1.6B, 3B model release

• v6 Finch: 7B, 14B training, this would be an upcycle of the Eagle models

• MoE: (approximately) 8 x 22B

Basically newer, better, bigger models - as we keep iterating on our goal to build a multi-lingual GPT4 class model, in open-source space, that can run on commodity hardware.

And ensure AI is accessible to everyone, regardless of language, or economic status.

Acknowledgment

We are grateful and would like to thank the following key groups:

• Recursal.ai team for financing the GPU resources, and managing the training of this foundation model - you can run the Eagle line of RWKV models on their cloud / on-premise platform today.

• EleutherAI for their support, especially in the v5/v6 Eagle/Finch paper

• Linux Foundation AI & Data group for supporting and hosting the RWKV project

Along with the various developers, working on the growing collection of RWKV-related projects.

We present the Eagle and Finch architecture paper at arxiv: https://arxiv.org/abs/2404.05892

Which covers and documents the architecture changes from RWKV-v4 onwards. This paper is a collaborative effort with the folks at Eleuther AI, who helped us in the paper-writing process

Special shout-out to

• BlinkDL: The creator of RWKV project

• Eleuther AI: Who helped us throughout the paper writing process

• Linux Foundation AI & Data: For hosting our project

• Stability AI: Who sponsored the bulk of the compute, for the models covered.

Does this cover our latest model?

No - this covers our previously released Eagle and Finch line of models, trained up to 1.1T tokens

A reminder, that as a fully Open Source project, we release in the following sequence: Code, Weights, then the paper Not the other way around

Stay tuned for more details on our upcoming models this week

• Eagle: 2.25T 7B

• Finch: 2.5T 1.6B

(Some of you probably already know where to find it, if you search through our repos / discord)

Eagle 7B - in short

Eagle 7B is a 7.52B parameter model that:

• Built on the RWKV-v5 architecture
  (a linear transformer with 10-100x+ lower inference cost)

• Ranks as the world’s greenest 7B model (per token)

• Trained on 1.1 Trillion Tokens across 100+ languages

• Outperforms all 7B class models in multi-lingual benchmarks

• Approaches Falcon (1.5T), LLaMA2 (2T), Mistral (>2T?) level of performance in English evals

• Trade blows with MPT-7B (1T) in English evals

• All while being an “Attention-Free Transformer”

• Is a foundation model, with a very small instruct tune - further fine-tuning is required for various use cases!
  

We are releasing RWKV-v5 Eagle 7B, licensed as Apache 2.0 license, under the Linux Foundation, and can be used personally or commercially without restrictions

• Download from Huggingface, and use it anywhere (even locally)

• Use our reference pip inference package, or any other community inference options (Desktop App, RWKV.cpp, etc)

• Fine-tune using our Infctx trainer

• Try it online on Huggingface

• [Pending PR] Get it merged into Huggingface transformers!

Multi-Lingual Performance details

We performed multi-lingual performance across the following benchmarks: xLAMBDA, xStoryCloze, xWinograd, xCopa

Across a total of 23 languages

Most of these benchmarks cover common sense reasoning, in their respective languages. And show a huge overall jump in multi-lingual performance for RWKV v4-to-v5 architecture. And the v2 world dataset.

It should also be noted, that there is a lack of multi-lingual benchmarks, as the above covers approximately the top 23 languages.

This makes it hard to evaluate model language performance directly over the remaining 75+ languages, over the total 100+ trained languages. A shortcoming we hope to improve in future models.

English Performance details

English performance was measured across 12 separate benchmarks, across commonsense reasoning, and world knowledge

Once again we see a huge overall jump from RWKV v4-to-v5 architecture. And the v2 world dataset.

Where v4 previously lost out to MPT-7b, the top model in the 1T token tier.

v5 begins trading blows in benchmarks, in some cases even coming on top in certain benchmarks ( LAMBADA, StoryCloze16, WinoGrande, HeadQA_en, Sciq ) over Falcon, or even llama2.

In addition, v5 performance starts to fall in line with the expected transformer performance level, with its given approximate token training count.

With Mistral-7B maintaining its lead with its rumored 2~7 Trillion token training.

We expect to narrow the gap, as we train an additional 1T token, to cross the llama2 line and hopefully reach the mistral line.

Alternatively, as a base model, which is lightly tuned (really small instruct set mixed in), we are eager to see how the various community and instruct-tuned variants

Perhaps a good dataset + Scalable architecture:
is all you need?

A notable observation was that our checkpoints near the 300 Billion token point, show similar performance to pythia-6.9b

This is consistent with previous pile-based experiments on our RWKV-v4 architecture, that linear transformers like RWKV scale similarly in performance levels to transformers, with the same token count training.

If so, it does repeat the question. If the exact architecture, matter less than the data for the model eval performance?

If true, perhaps we should seek more efficient and scalable architecture, to increase accessibility, drive the cost of AI downwards for everyone, and lessen the impact on our environment.

Building inclusive AI for everyone in this world - not just the English

A common feedback we receive for the RWKV multi-lingual approach is

• it hurts our English evaluation scores and slows the growth of linear transformers

• that it is not fair to compare the multi-lingual performance of a multi-lingual model -vs- a purely English model

And for most parts, we agree on both points.

But we have no plans on changing this, as we are building AI for the world - which is not just an English world.

In 2023, only 17% of the world's population speaks English
( 1.3 billion people )

However, by ensuring support for the top 25 languages in the world and beyond, we can cover approximately 4 billion people, or 50% of the world

This aligns well with the team’s common goal, of getting AI to support everyone, not just by allowing it to run cheaply and affordably even on lower-end hardware. But by supporting their language.

Over time, we intend to grow the multi-lingual dataset, to support a wider variety of languages, and to slowly grow that coverage to 100% of the world - to ensure no language gets left behind.

A major example of this in our community is the Indonesian-NLP discord group, which finetunes an Indonesian language model from the RWKV line of base models.

Allowing them to build strong language-specific models - on a cheap affordable basis (ie. single node), without needing to do half a million dollars of pre-training.

Future Plans

This release marks the release of the strongest linear transformer (in terms of eval benchmarks) to date.

While it may not have succeeded in passing LLaMA2 and Mistral. It provides strong evidence of the following

• The RWKV-v5 model architecture scales similarly to transformer performance with a similar token count

• You can achieve a near LLaMA2-like level of performance, with a substantially lower inference cost

• While supporting multi-lingual levels of performance
  

We plan to follow by pushing further ahead with

• [Feb 2024] An updated RWKV v5: Eagle paper, where we will go deeper in-depth on the architecture changes since v4, and the model benchmarks and evals

• [Feb 2024] A further 1T token in training (2T total), for direct comparisons with the LLaMA2 7B model

• [Mar 2024] An MoE model based on the v5 Eagle 2T model

• [Mar 2024] RWKV-v6: “Finch” 1.5B, 3B world models

Disclaimer: All dates are approximate, and is heavily subjected to compute avaliability from our sponsors/provider

Find more about the RWKV Project at

• Wiki: https://wiki.rwkv.com/

• Discord: https://discord.gg/bDSBUMeFpc

Acknowledgment

We are grateful and would like to thank the following key groups:

• StabilityAI for the bulk of the computing provided to train this foundation model

• EleutherAI for their support, especially in the ongoing paper-writing process

• Linux Foundation AI & Data group for supporting and hosting the RWKV project

Along with the various developers, working on the growing collection of RWKV-related projects.

In that regard, we are proud to say that RWKV (the model which our team works on), has been independently benchmarked as the world's greenest and most energy-efficient AI model/architecture, on a per token output basis, for models of the same param sizes (7B params).

The energy efficiency of the RWKV architecture is derived from the 10-100 times compute efficiency of our linear transformer architecture vs the quadratic scaling of transformer architectures. A benefit we expect to scale better as our models get large

Combined with how RWKV models scales similarly to transformers in evals, against other models with the same dataset.

The industry wide benefits for scaling more energy efficient architecture, like RWKV, will be significant for our industry as a whole.

Towards a future with more, not less alternatives to AI,
with the various unique benefits each architecture will bring us.

Repost note: This is a repost of a past blogpost, prior to the setup of this blog

RWKV v5 1.5B achieves SOTA status with

• Industry leading multi-lingual performance (across xLBD, xSC, xWG, xCOPA benchmarks) by significant margins, against all existing models

• Comparable performance to falcon-rw-1b in english based benchmark

  • We win out in LAMBDA, StoryCloze16, arch_challenge, arc_easy, headQA_en, openbookQA, sciq, COPA

  • but looses out very slightly on PIQA, Hellaswag, WinoGrade,ReCoRD, COPA

For nearly all use cases under the 2B param model class, RWKV V5 now represents either the best model for multi-lingual use, or a tied 1st place model with falcon-rw-1b

Making this a strong default model of choice within its weight class.

A pattern we intend to repeat in the 3, 7, and 14B weight classes respectively. We expect the 3B model to be out by first week december.

You can access the model today via the following options

• Public Demo: https://huggingface.co/spaces/BlinkDL/ChatRWKV-gradio

• Model Download : https://huggingface.co/BlinkDL/rwkv-5-world/tree/main

This is a repost of a past event, prior to the setup of this blog

Ensuring that RWKV continues to grow as a true OSS model (Just Apache 2 license) By the community, for the world

Thanks @LFAIDataFdn for welcoming us on board OSS summit

More information can be found here on the official press briefing:
https://lfaidata.foundation/blog/2023/09/21/lf-ai-data-launches-generative-ai-commons/

This is a repost of a past event, prior to the setup of this blog