Envoy Remote Access Log Collector
Envoy proxy logs contain some information about our request that’s is useful for debugging and troubleshooting. Envoy proxies print access info to stdout.
- Enable Envoy’s access logging with istio
You can also choose between JSON and text by setting accessLogEncoding to JSON or TEXT.
You may also want to customize the format of the access log by editing accessLogFormat.
spec:
meshConfig:
accessLogFile: /dev/stdout
accessLogEncoding: JSON- Enable per workload access log
Use EnvoyFilter to modify values for certain fields, add specific filters, or even add entirely new listeners, clusters, etc.
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: searchapi-access-log
spec:
workloadSelector:
labels:
app: search-api
configPatches:
- applyTo: NETWORK_FILTER
match:
context: ANY
listener:
filterChain:
filter:
name: "envoy.http_connection_manager"
patch:
operation: MERGE
value:
typed_config:
"@type": "type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager"
access_log:
- name: envoy.file_access_log
config:
path: /dev/stdout
jsonFormat:
authority: "%REQ(:AUTHORITY)%"
path: "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%"
protocol: "%PROTOCOL%"
response_code: "%RESPONSE_CODE%"
response_flags: "%RESPONSE_FLAGS%"
route_name: "%ROUTE_NAME%"
upstream_host: "%UPSTREAM_HOST%"
user_agent: "%REQ(USER-AGENT)%"
x_agent_name: "%REQ(X-AGENTNAME)%"
x_forwarded_for: "%REQ(X-FORWARDED-FOR)%"How do we send to access log remote service?
We need to build a grpc server. with an endpoint that responds to AccessLogService_StreamAccessLogsServer. Envoy then will connect to it to send the logs. At his point envoyproxy go-control-plane help us to build our remote access log service.
Register our grpc server to Envoy. Envoy will connect and send StreamAccessLogsMessage messages forever. And transform HTTP or TCP logs to our current model.
grpcServer := grpc.NewServer()
//Acl log
v2.RegisterAccessLogServiceServer(grpcServer, converter.New())We listen to grpc access log stream forever.
func (s *server) StreamAccessLogs(stream v2.AccessLogService_StreamAccessLogsServer) error {
for {
in, err := stream.Recv()
if err == io.EOF {
return nil
}
if err != nil {
return err
}
logEntries := transform(in.GetHttpLogs())
for _, logEntry := range logEntries {
var data []byte
data, err = json.Marshal(logEntry)
fmt.Println(string(data))
}
}
}We need to configure istio control plane for sending access log to our collector.
data:
mesh: |-
accessLogEncoding: JSON
accessLogFile: ""
accessLogFormat: ""
connectTimeout: 15s
enableEnvoyAccessLogService: true
defaultConfig:
envoyAccessLogService:
address: envoy-accesslog-collector.platform:9001enableEnvoyAccessLogService: This flag enables Envoy’s gRPC Access Log Service.
envoyAccessLogService.address: Address of the service to which access logs from Envoys should be sent.
2021-04-17T09:29:51.252863378Z {"timestamp":"","traceId":"6e3ead4b-3de3-4633-99ad-c961a00c74fa","method":"GET","protocol":"HTTP11","scheme":"","statusCode":503,"domain":"search","path":"/available-indexes","userAgent":"Apache-HttpClient/4.5.10 (Java/1.8.0_222)","referer":"","forwardedFor":"x","upsteamFailure":"","upstreamCluster":"","downstreamDirectRemoteAddress":{"address":"x","port":21288},"downstreamRemoteAddress":{"address":"x","port":21288},"downstreamLocalAddress":{"address":"x","port":8080},"upstreamRemoteAddress":{"address":"","port":0},"upstreamLocalAddress":{"address":"","port":0},"time_ms":0,"request":{"headersBytes":1498,"bodyBytes":0,"remoteIP":"x"},"response":{"details":"no_healthy_upstream","headersBytes":113,"bodyBytes":19}}See full code in here.
References:
