Sessi 2 — Tokenisasi & Normalisasi (Pra‑Pemrosesan Teks)

Tujuan: merancang pipeline pra‑proses yang reproducible, mengurangi noise (URL, emoji, angka, elongasi), dan menyiapkan data untuk TF–IDF/embedding pada sesi berikutnya.

2) Dataset (≥ 30 Dokumen Campuran ID+EN)

Pakai korpus multi‑domain yang lebih noisy agar aturan normalisasi berdampak nyata. Anda boleh menambah dokumen kelas sendiri.

corpus = [
  "Pengiriman 2x lebih cepat dr ekspetasi!!! link: https://toko.id/a/123 😍",
  "Battery-life ~10hrs; screen-glare outdoors :( #annoying",
  "Pelayanan CS oke bgt, tp refund > 7 hari ya? email: cs@brand.co",
  "This movie is sooo goooood!!! Must watch!!!",
  "Harga 1.299.000 jd 1.099.000 pas promo 11.11 😱",
  "Server down @ 02:13 UTC; logs @ s3://bucket/app/logs",
  "Rasa kopi mantapppp!! ☕️ kemasan rapi, ekspedisi cepat.",
  "Update v2.1 fixed crash; but login delay ~3-5s remains.",
  "UI/UX oke, on-boarding jelas; dark-mode 👍",
  "Dokumennya lengkap bgt → examples jelas, thanks!",
  "Packing kurang rapih; item aman sih. Kurir telat 1 hari.",
  "Support promised callback @Mon, never happened...",
  "Keyboard travel enakkkk; backlight ok; keycaps anti-slip.",
  "Kualitas kain adem bgt; ukuran pas; jahitan rapiii~",
  "Bluetooth stable ~8m; beyond that stutter.",
  "Staff ramah; kamar wangi? hmm bau rokok dikit :/",
  "Checkout gagal (PG timeouts). Retry 3x baru bisa.",
  "Harga murah; performa ok buat mahasiswa.",
  "Stabilisasi video kurang; night-mode mantap.",
  "Delivery was late; packaging dented; item OK tho.",
  "Resep gampang; 20' jadi. Anak2 suka.",
  "Top-ups via e-wallet failed 2x; SMS OTP telat.",
  "Refund diproses <24h setelah komplain. mantap!",
  "Weather ingestion needs tz normalization (Asia/Jakarta).",
  "Docs thorough; examples accelerate onboarding.",
  "Price/perf = excellent; would rec to classmates.",
  "App cache clear helps; after update still slow :(",
  "Kopi pait? menurutku balanced kok ☺",
  "Layar AMOLED tajam; nits tinggi; glare minim.",
  "Shipping fee too high for small items..."
]

3) Praktik Google Colab — Pipeline Pra‑Proses

Salin per blok ke Colab. Setiap bagian independen dan bisa dieksekusi berurutan.

A. Setup

!pip -q install pandas numpy scikit-learn nltk regex emoji Unidecode

import re, regex as re2, math, unicodedata
import numpy as np, pandas as pd
import matplotlib.pyplot as plt
from collections import Counter
from emoji import replace_emoji
from unidecode import unidecode
import nltk
nltk.download('punkt', quiet=True)

pd.set_option('display.max_colwidth', 120)

# Muat korpus
# (Anda dapat impor dari CSV: pd.read_csv('corpus_sessi1.csv')['text'].tolist())

B. Aturan Normalisasi yang Dapat Dikonfigurasi

# Placeholder standar
PLACEHOLDERS = {"URL":"", "EMAIL":"", "USER":"", "NUM":"", "EMOJI":""}

URL_RE   = re2.compile(r"https?://\S+|s3://\S+", re2.I)
EMAIL_RE = re2.compile(r"[\w.+-]+@[\w-]+\.[\w.-]+", re2.I)
USER_RE  = re2.compile(r"@\w+")
NUM_RE   = re2.compile(r"(?

def tokenize_regex(text):
    return [m.group(0) for m in TOKEN_RE.finditer(text)]

# NLTK (umum); untuk ID bisa tetap pakai regex custom
from nltk.tokenize import word_tokenize

toks_raw = [tokenize_regex(t) for t in corpus]

toks_norm_regex = [tokenize_regex(t) for t in norm]

toks_norm_nltk  = [word_tokenize(t) for t in norm]

print("Contoh token RAW  :", toks_raw[0][:12])
print("Contoh token NORM R", toks_norm_regex[0][:12])
print("Contoh token NORM N", toks_norm_nltk[0][:12])

def vocab_size(tokens_list):
    return len(set(tok for doc in tokens_list for tok in doc))

def ttr(tokens_list):
    total = sum(len(doc) for doc in tokens_list)
    return vocab_size(tokens_list) / max(total,1)

raw_V, norm_V = vocab_size(toks_raw), vocab_size(toks_norm_regex)
raw_TTR, norm_TTR = ttr(toks_raw), ttr(toks_norm_regex)

print({"raw_vocab": raw_V, "norm_vocab": norm_V, "raw_TTR": raw_TTR, "norm_TTR": norm_TTR})

# Top 25 kata sesudah normalisasi
from collections import Counter
freq_norm = Counter([t for d in toks_norm_regex for t in d])
TOP = pd.DataFrame(freq_norm.most_common(25), columns=["term","freq"]) 
print(TOP)

plt.figure(figsize=(10,5))
plt.bar(TOP["term"], TOP["freq"])
plt.xticks(rotation=60, ha='right')
plt.title('Top 25 Istilah setelah Normalisasi')
plt.tight_layout(); plt.show()

sample_ix = [0,1,3,4,10,14,18,22]
view = pd.DataFrame({
    "original": [corpus[i] for i in sample_ix],
    "normalized": [norm[i] for i in sample_ix]
})
print(view)

pd.DataFrame({"text": norm}).to_csv("corpus_sessi2_normalized.csv", index=False)
print("Tersimpan: corpus_sessi2_normalized.csv")

# Jika tugas Anda butuh angka real (mis. kuantifikasi), matikan mask_num=True.
# Jika sentimen bergantung pada emoji, pertahankan EMOJI sebagai token, bukan placeholder.