구글, 개방형 AI 모델로 급선회… 폐쇄형 오픈AI와 ‘진검승부’

https://v.daum.net/v/20240223050328046

 

구글, 개방형 AI 모델로 급선회… 폐쇄형 오픈AI와 ‘진검승부’

생성형 인공지능(AI) 업계에서 오픈AI와 패권을 다투는 구글이 ‘오픈소스(개방형) AI’ 진영으로 한 걸음 이동했다. 업계는 챗GPT의 기반 모델인 ‘GPT4’의 어떤 것도 공개하지 않는 오픈AI와 구글

v.daum.net

 

 

구글은 21일(현지시간) 거대언어모델(LLM) ‘젬마’를 오픈소스로 공개했다. 이에 따라 개별 연구자나 개발자, 기업, 연구기관 등은 젬마를 자유롭게 활용할 수 있다. 젬마는 구글의 AI 모델 ‘제미나이’의 경량형이라고 생각하면 쉽다.

 

>

제미나이 아주 잘 쓰고 있음.

다음 목표는 젬마!

(그 다음 라마, 버트. 솔라는 했고.)

Posted by 캬웃
,

[코드]

###########################################
# 2-1. Zero-shot 평가 (영어)
## About hellaswag, copa, boolq, mmlu

!lm_eval --model hf \
    --model_args pretrained=[...Custom_LLM...] \
    --tasks hellaswag,copa,boolq,mmlu \
    --device cuda:0 \
    --batch_size 8 \
    --num_fewshot 0

 

 

[결과]

hf (pretrained=cashbook/SOLAR-Platypus-10.7B-v1-kjw), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: 8
|                 Tasks                 |Version|Filter|n-shot| Metric |Value |   |Stderr|
|---------------------------------------|-------|------|-----:|--------|-----:|---|-----:|
|mmlu                                   |N/A    |none  |     0|acc     |0.6304|±  |0.0038|
| - humanities                          |N/A    |none  |     0|acc     |0.5626|±  |0.0066|
|  - formal_logic                       |      0|none  |     0|acc     |0.3413|±  |0.0424|
|  - high_school_european_history       |      0|none  |     0|acc     |0.7818|±  |0.0323|
|  - high_school_us_history             |      0|none  |     0|acc     |0.8284|±  |0.0265|
|  - high_school_world_history          |      0|none  |     0|acc     |0.8101|±  |0.0255|
|  - international_law                  |      0|none  |     0|acc     |0.8099|±  |0.0358|
|  - jurisprudence                      |      0|none  |     0|acc     |0.7407|±  |0.0424|
|  - logical_fallacies                  |      0|none  |     0|acc     |0.7607|±  |0.0335|
|  - moral_disputes                     |      0|none  |     0|acc     |0.7312|±  |0.0239|
|  - moral_scenarios                    |      0|none  |     0|acc     |0.2413|±  |0.0143|
|  - philosophy                         |      0|none  |     0|acc     |0.7074|±  |0.0258|
|  - prehistory                         |      0|none  |     0|acc     |0.7500|±  |0.0241|
|  - professional_law                   |      0|none  |     0|acc     |0.4831|±  |0.0128|
|  - world_religions                    |      0|none  |     0|acc     |0.8129|±  |0.0299|
| - other                               |N/A    |none  |     0|acc     |0.7219|±  |0.0077|
|  - business_ethics                    |      0|none  |     0|acc     |0.7000|±  |0.0461|
|  - clinical_knowledge                 |      0|none  |     0|acc     |0.6981|±  |0.0283|
|  - college_medicine                   |      0|none  |     0|acc     |0.6474|±  |0.0364|
|  - global_facts                       |      0|none  |     0|acc     |0.3600|±  |0.0482|
|  - human_aging                        |      0|none  |     0|acc     |0.7175|±  |0.0302|
|  - management                         |      0|none  |     0|acc     |0.7961|±  |0.0399|
|  - marketing                          |      0|none  |     0|acc     |0.8932|±  |0.0202|
|  - medical_genetics                   |      0|none  |     0|acc     |0.7800|±  |0.0416|
|  - miscellaneous                      |      0|none  |     0|acc     |0.8340|±  |0.0133|
|  - nutrition                          |      0|none  |     0|acc     |0.7516|±  |0.0247|
|  - professional_accounting            |      0|none  |     0|acc     |0.5319|±  |0.0298|
|  - professional_medicine              |      0|none  |     0|acc     |0.7022|±  |0.0278|
|  - virology                           |      0|none  |     0|acc     |0.5241|±  |0.0389|
| - social_sciences                     |N/A    |none  |     0|acc     |0.7423|±  |0.0077|
|  - econometrics                       |      0|none  |     0|acc     |0.4737|±  |0.0470|
|  - high_school_geography              |      0|none  |     0|acc     |0.8131|±  |0.0278|
|  - high_school_government_and_politics|      0|none  |     0|acc     |0.8756|±  |0.0238|
|  - high_school_macroeconomics         |      0|none  |     0|acc     |0.6308|±  |0.0245|
|  - high_school_microeconomics         |      0|none  |     0|acc     |0.7269|±  |0.0289|
|  - high_school_psychology             |      0|none  |     0|acc     |0.8367|±  |0.0158|
|  - human_sexuality                    |      0|none  |     0|acc     |0.7786|±  |0.0364|
|  - professional_psychology            |      0|none  |     0|acc     |0.6667|±  |0.0191|
|  - public_relations                   |      0|none  |     0|acc     |0.7000|±  |0.0439|
|  - security_studies                   |      0|none  |     0|acc     |0.7388|±  |0.0281|
|  - sociology                          |      0|none  |     0|acc     |0.8507|±  |0.0252|
|  - us_foreign_policy                  |      0|none  |     0|acc     |0.8600|±  |0.0349|
| - stem                                |N/A    |none  |     0|acc     |0.5322|±  |0.0086|
|  - abstract_algebra                   |      0|none  |     0|acc     |0.3300|±  |0.0473|
|  - anatomy                            |      0|none  |     0|acc     |0.5926|±  |0.0424|
|  - astronomy                          |      0|none  |     0|acc     |0.7039|±  |0.0372|
|  - college_biology                    |      0|none  |     0|acc     |0.7708|±  |0.0351|
|  - college_chemistry                  |      0|none  |     0|acc     |0.4100|±  |0.0494|
|  - college_computer_science           |      0|none  |     0|acc     |0.5400|±  |0.0501|
|  - college_mathematics                |      0|none  |     0|acc     |0.3800|±  |0.0488|
|  - college_physics                    |      0|none  |     0|acc     |0.4118|±  |0.0490|
|  - computer_security                  |      0|none  |     0|acc     |0.7300|±  |0.0446|
|  - conceptual_physics                 |      0|none  |     0|acc     |0.5362|±  |0.0326|
|  - electrical_engineering             |      0|none  |     0|acc     |0.5655|±  |0.0413|
|  - elementary_mathematics             |      0|none  |     0|acc     |0.4339|±  |0.0255|
|  - high_school_biology                |      0|none  |     0|acc     |0.7742|±  |0.0238|
|  - high_school_chemistry              |      0|none  |     0|acc     |0.4975|±  |0.0352|
|  - high_school_computer_science       |      0|none  |     0|acc     |0.6200|±  |0.0488|
|  - high_school_mathematics            |      0|none  |     0|acc     |0.3593|±  |0.0293|
|  - high_school_physics                |      0|none  |     0|acc     |0.3974|±  |0.0400|
|  - high_school_statistics             |      0|none  |     0|acc     |0.5509|±  |0.0339|
|  - machine_learning                   |      0|none  |     0|acc     |0.4286|±  |0.0470|
|hellaswag                              |      1|none  |     0|acc     |0.6396|±  |0.0048|
|                                       |       |none  |     0|acc_norm|0.8310|±  |0.0037|
|copa                                   |      1|none  |     0|acc     |0.8700|±  |0.0338|
|boolq                                  |      2|none  |     0|acc     |0.8260|±  |0.0066|

|      Groups      |Version|Filter|n-shot|Metric|Value |   |Stderr|
|------------------|-------|------|-----:|------|-----:|---|-----:|
|mmlu              |N/A    |none  |     0|acc   |0.6304|±  |0.0038|
| - humanities     |N/A    |none  |     0|acc   |0.5626|±  |0.0066|
| - other          |N/A    |none  |     0|acc   |0.7219|±  |0.0077|
| - social_sciences|N/A    |none  |     0|acc   |0.7423|±  |0.0077|
| - stem           |N/A    |none  |     0|acc   |0.5322|±  |0.0086|

 

'LLM' 카테고리의 다른 글

BART (Bidirectional Auto-Regressive Transformer)  (0) 2024.02.20
[에러] config.json  (0) 2024.02.20
Posted by 캬웃
,

BERT(Bidirectional Encoder Representations from Transformers)는 구글 연구원이 2018년에 도입한 마스킹된 언어 모델 제품군이다.

Posted by 캬웃
,