Talking specifically about your example you will need to paste your .behavior definition part under the .spec like below:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: nginx
spec:
  # <--- START ---> 
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 100
        periodSeconds: 15
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
      - type: Percent
        value: 100
        periodSeconds: 15
      - type: Pods
        value: 4
        periodSeconds: 15
      selectPolicy: Max
      # <--- END ---> 
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: nginx
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 50
  - type: Resource
    resource:
      name: memory
      target:
        type: AverageValue
        averageValue: 100Mi

Please remember that this feature is available from Kubernetes v1.18.

Earlier version of Kubernetes will show following error:

error: error validating "hpa.yaml": error validating data: ValidationError(HorizontalPodAutoscaler.spec): unknown field "behavior" in io.k8s.api.autoscaling.v2beta2.HorizontalPodAutoscalerSpec; if you choose to ignore these errors, turn validation off with --validate=false

As for a side note you can also take a look on:

• $ kubectl autoscale
• $ kubectl autoscale deployment nginx --min=1 --max=10 --cpu-percent=80 <- example

Creates an autoscaler that automatically chooses and sets the number of pods that run in a kubernetes cluster.

Additional reference:

• Kubernetes.io: Docs: Tasks: Run application: Horizontal pod autoscaler
• Kubernetes.io: Docs: Reference: Generated: Kubectl: kubectl commands: autoscale